From smarks at openjdk.java.net Thu Oct 1 00:21:16 2020 From: smarks at openjdk.java.net (Stuart Marks) Date: Thu, 1 Oct 2020 00:21:16 GMT Subject: RFR: 8156071: List.of: reduce array copying during creation Message-ID: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Plumb new internal static factory method to trust the array passed in, avoiding unnecessary copying. JMH results for the benchmark show about 15% improvement for the cases that were optimized, namely the 3 to 10 fixed arg cases. # VM options: -verbose:gc -XX:+UseParallelGC -Xms4g -Xmx4g --enable-preview -verbose:gc -XX:+UsePara llelGC -Xms4g -Xmx4g -Xint # Warmup: 5 iterations, 1 s each # Measurement: 5 iterations, 2 s each WITHOUT varargs optimization: Benchmark Mode Cnt Score Error Units ListArgs.list00 thrpt 15 6019.539 ? 144.040 ops/ms ListArgs.list01 thrpt 15 1985.009 ? 40.606 ops/ms ListArgs.list02 thrpt 15 1854.812 ? 17.488 ops/ms ListArgs.list03 thrpt 15 963.866 ? 10.262 ops/ms ListArgs.list04 thrpt 15 908.116 ? 6.278 ops/ms ListArgs.list05 thrpt 15 848.607 ? 16.701 ops/ms ListArgs.list06 thrpt 15 822.282 ? 8.905 ops/ms ListArgs.list07 thrpt 15 780.057 ? 11.214 ops/ms ListArgs.list08 thrpt 15 745.295 ? 19.204 ops/ms ListArgs.list09 thrpt 15 704.596 ? 14.003 ops/ms ListArgs.list10 thrpt 15 696.436 ? 4.914 ops/ms ListArgs.list11 thrpt 15 661.908 ? 11.041 ops/ms WITH varargs optimization: Benchmark Mode Cnt Score Error Units ListArgs.list00 thrpt 15 6172.298 ? 62.736 ops/ms ListArgs.list01 thrpt 15 1987.724 ? 45.468 ops/ms ListArgs.list02 thrpt 15 1843.419 ? 10.693 ops/ms ListArgs.list03 thrpt 15 1126.946 ? 30.952 ops/ms ListArgs.list04 thrpt 15 1050.440 ? 17.859 ops/ms ListArgs.list05 thrpt 15 999.275 ? 23.656 ops/ms ListArgs.list06 thrpt 15 948.844 ? 19.615 ops/ms ListArgs.list07 thrpt 15 897.541 ? 15.531 ops/ms ListArgs.list08 thrpt 15 853.359 ? 18.755 ops/ms ListArgs.list09 thrpt 15 826.394 ? 8.284 ops/ms ListArgs.list10 thrpt 15 779.231 ? 4.104 ops/ms ListArgs.list11 thrpt 15 650.888 ? 3.948 ops/ms ------------- Commit messages: - 8156071: reduce varargs array creation and unnecessary copying during creation Changes: https://git.openjdk.java.net/jdk/pull/449/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=449&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8156071 Stats: 203 lines in 6 files changed: 185 ins; 0 del; 18 mod Patch: https://git.openjdk.java.net/jdk/pull/449.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/449/head:pull/449 PR: https://git.openjdk.java.net/jdk/pull/449 From valeriep at openjdk.java.net Thu Oct 1 02:27:25 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Thu, 1 Oct 2020 02:27:25 GMT Subject: RFR: 8249783: Simplify DerValue and DerInputStream [v3] In-Reply-To: References: Message-ID: On Tue, 29 Sep 2020 15:20:24 GMT, Weijun Wang wrote: >> This code change rewrites DerValue into a mostly immutable class and simplifies DerInputStream as a wrapper for a >> series of DerValues objects. DerInputBuffer is removed. >> All existing methods of DerValue and DerInputStream should still work with the exact same behavior, except for a few >> places where bugs are fixed. For example, Indefinite length must be used with a constructed tag. >> Except for the ObjectIdentifier class where DerInputBuffer is directly referenced, no other code is touched. > > Weijun Wang has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev > excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since > the last revision: > - 8249783: Simplify DerValue and DerInputStream > - Merge remote-tracking branch 'origin/master' into 8249783 > - Enhance DerValue::getOctetString to be able to read multi-level constructed value. > - 8249783: Simplify DerValue and DerInputStream Updated webrev looks fine. Only a minor nit regarding a comment remaining. ------------- Marked as reviewed by valeriep (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/232 From weijun at openjdk.java.net Thu Oct 1 03:40:26 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 1 Oct 2020 03:40:26 GMT Subject: RFR: 8249783: Simplify DerValue and DerInputStream [v4] In-Reply-To: References: Message-ID: > This code change rewrites DerValue into a mostly immutable class and simplifies DerInputStream as a wrapper for a > series of DerValues objects. DerInputBuffer is removed. > All existing methods of DerValue and DerInputStream should still work with the exact same behavior, except for a few > places where bugs are fixed. For example, Indefinite length must be used with a constructed tag. > Except for the ObjectIdentifier class where DerInputBuffer is directly referenced, no other code is touched. Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: some small tuning ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/232/files - new: https://git.openjdk.java.net/jdk/pull/232/files/ac77172e..23205637 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=232&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=232&range=02-03 Stats: 26 lines in 2 files changed: 4 ins; 11 del; 11 mod Patch: https://git.openjdk.java.net/jdk/pull/232.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/232/head:pull/232 PR: https://git.openjdk.java.net/jdk/pull/232 From weijun at openjdk.java.net Thu Oct 1 03:42:59 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 1 Oct 2020 03:42:59 GMT Subject: RFR: 8249783: Simplify DerValue and DerInputStream [v3] In-Reply-To: References: Message-ID: On Thu, 1 Oct 2020 02:24:47 GMT, Valerie Peng wrote: >> Weijun Wang has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev >> excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since >> the last revision: >> - 8249783: Simplify DerValue and DerInputStream >> - Merge remote-tracking branch 'origin/master' into 8249783 >> - Enhance DerValue::getOctetString to be able to read multi-level constructed value. >> - 8249783: Simplify DerValue and DerInputStream > > Updated webrev looks fine. Only a minor nit regarding a comment remaining. Added a new commit. Mostly comment or style change. Several always-false `if (lenByte < 0)` check removed. `result.toArray(new DerValue[0])` used because new knowledge shows it's faster than `result.toArray(new DerValue[result.size()])`. `getSet(int,boolean implicit)` added check for `implicit` to be semantically correct (although it's always called with `implicit == true`. ------------- PR: https://git.openjdk.java.net/jdk/pull/232 From github.com+232002+martin-g at openjdk.java.net Thu Oct 1 06:29:58 2020 From: github.com+232002+martin-g at openjdk.java.net (Martin Grigorov) Date: Thu, 1 Oct 2020 06:29:58 GMT Subject: RFR: 8156071: List.of: reduce array copying during creation In-Reply-To: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> References: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Message-ID: On Thu, 1 Oct 2020 00:13:28 GMT, Stuart Marks wrote: > Plumb new internal static factory method to trust the array passed in, avoiding unnecessary copying. JMH results for > the benchmark show about 15% improvement for the cases that were optimized, namely the 3 to 10 fixed arg cases. > # VM options: -verbose:gc -XX:+UseParallelGC -Xms4g -Xmx4g --enable-preview -verbose:gc -XX:+UsePara > llelGC -Xms4g -Xmx4g -Xint > # Warmup: 5 iterations, 1 s each > # Measurement: 5 iterations, 2 s each > > WITHOUT varargs optimization: > > Benchmark Mode Cnt Score Error Units > ListArgs.list00 thrpt 15 6019.539 ? 144.040 ops/ms > ListArgs.list01 thrpt 15 1985.009 ? 40.606 ops/ms > ListArgs.list02 thrpt 15 1854.812 ? 17.488 ops/ms > ListArgs.list03 thrpt 15 963.866 ? 10.262 ops/ms > ListArgs.list04 thrpt 15 908.116 ? 6.278 ops/ms > ListArgs.list05 thrpt 15 848.607 ? 16.701 ops/ms > ListArgs.list06 thrpt 15 822.282 ? 8.905 ops/ms > ListArgs.list07 thrpt 15 780.057 ? 11.214 ops/ms > ListArgs.list08 thrpt 15 745.295 ? 19.204 ops/ms > ListArgs.list09 thrpt 15 704.596 ? 14.003 ops/ms > ListArgs.list10 thrpt 15 696.436 ? 4.914 ops/ms > ListArgs.list11 thrpt 15 661.908 ? 11.041 ops/ms > > WITH varargs optimization: > > Benchmark Mode Cnt Score Error Units > ListArgs.list00 thrpt 15 6172.298 ? 62.736 ops/ms > ListArgs.list01 thrpt 15 1987.724 ? 45.468 ops/ms > ListArgs.list02 thrpt 15 1843.419 ? 10.693 ops/ms > ListArgs.list03 thrpt 15 1126.946 ? 30.952 ops/ms > ListArgs.list04 thrpt 15 1050.440 ? 17.859 ops/ms > ListArgs.list05 thrpt 15 999.275 ? 23.656 ops/ms > ListArgs.list06 thrpt 15 948.844 ? 19.615 ops/ms > ListArgs.list07 thrpt 15 897.541 ? 15.531 ops/ms > ListArgs.list08 thrpt 15 853.359 ? 18.755 ops/ms > ListArgs.list09 thrpt 15 826.394 ? 8.284 ops/ms > ListArgs.list10 thrpt 15 779.231 ? 4.104 ops/ms > ListArgs.list11 thrpt 15 650.888 ? 3.948 ops/ms src/java.base/share/classes/jdk/internal/access/SharedSecrets.java line 88: > 86: if (javaUtilCollectionAccess == null) { > 87: try { > 88: Class.forName("java.util.ImmutableCollections$Access", true, null); How does this work ? It attempts to load this class but `javaUtilCollectionAccess` is never assigned to a new value. **Update**: I just noticed this is the getter. ------------- PR: https://git.openjdk.java.net/jdk/pull/449 From jpai at openjdk.java.net Thu Oct 1 14:42:21 2020 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 1 Oct 2020 14:42:21 GMT Subject: RFR: 8242882: opening jar file with large manifest might throw NegativeArraySizeException [v3] In-Reply-To: References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> Message-ID: On Wed, 30 Sep 2020 18:38:40 GMT, Lance Andersen wrote: >> I think it's fine either way. > > If you are going to validate the message, which I probably would not, it would be important to make sure it document > if the message is changed in JarFile::getBytes, that the test needs updated. Otherwise it will be easy to miss. Hello Lance, I decided to remove the assertion on the exception message. I have updated the PR accordingly. ------------- PR: https://git.openjdk.java.net/jdk/pull/323 From jpai at openjdk.java.net Thu Oct 1 14:42:21 2020 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 1 Oct 2020 14:42:21 GMT Subject: RFR: 8242882: opening jar file with large manifest might throw NegativeArraySizeException [v3] In-Reply-To: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> Message-ID: <9AgaGLA8q63mDTKDeavbyfZVntagI0bd0Kb7rkQQYyg=.67d4410c-b7db-4048-87d0-4ec1f9d93cd2@github.com> > Can I please get a review and a sponsor for a fix for https://bugs.openjdk.java.net/browse/JDK-8242882? > > As noted in that JBS issue, if the size of the Manifest entry in the jar happens to be very large (such that it exceeds > the `Integer.MAX_VALUE`), then the current code in `JarFile#getBytes` can lead to a `NegativeArraySizeException`. This > is due to the: if (len != -1 && len <= 65535) block which evaluates to `true` when the size of the manifest entry is > larger than `Integer.MAX_VALUE`. As a result, this then ends up calling the code which can lead to the > `NegativeArraySizeException`. The commit in this PR fixes that issue by changing those `if/else` blocks to prevent > this issue and instead use a code path that leads to the `InputStream#readAllBytes()` which internally has the > necessary checks to throw the expected `OutOfMemoryError`. This commit also includes a jtreg test case which > reproduces the issue and verifies the fix. Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: Second round of review comments addressed ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/323/files - new: https://git.openjdk.java.net/jdk/pull/323/files/279c7c83..a011b0d6 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=323&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=323&range=01-02 Stats: 34 lines in 2 files changed: 5 ins; 15 del; 14 mod Patch: https://git.openjdk.java.net/jdk/pull/323.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/323/head:pull/323 PR: https://git.openjdk.java.net/jdk/pull/323 From jpai at openjdk.java.net Thu Oct 1 14:42:24 2020 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 1 Oct 2020 14:42:24 GMT Subject: RFR: 8242882: opening jar file with large manifest might throw NegativeArraySizeException [v2] In-Reply-To: References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> Message-ID: On Wed, 30 Sep 2020 17:21:14 GMT, Brent Christian wrote: >> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: >> >> Address the review comments and introduce an array size check in JarFile.getBytes() method itself > > src/java.base/share/classes/java/util/jar/JarFile.java line 161: > >> 159: * OutOfMemoryError: Required array size too large >> 160: */ >> 161: private static final int MAX_BUFFER_SIZE = Integer.MAX_VALUE - 8; > > "/**" comments are generally only used for public documentation. For use here, probably a single line // comment would > be sufficient to explain what this value is. > This constant is also named, "MAX_ARRAY_SIZE" in various places, which seems more applicable to this case. Hello Brent, I've updated the PR with your suggested changes for this member variable name and the comment. > src/java.base/share/classes/java/util/jar/JarFile.java line 801: > >> 799: if (len > MAX_BUFFER_SIZE) { >> 800: throw new OutOfMemoryError("Required array size too large"); >> 801: } > > I would just add a new `long zeSize` to read and check `ze.getSize()`, and then (int) cast it into `len`, as before. > Then I think no changes would be needed past L802, `int bytesRead;` Done. Changed it based on your input. > test/jdk/java/util/jar/JarFile/LargeManifestOOMTest.java line 78: > >> 76: bw.write("OOM-Test: "); >> 77: for (long i = 0; i < 2147483648L; i++) { >> 78: bw.write("a"); > > As you probably noticed, this test takes a little while to run. One way to speed it up a little would be to write more > characters at a time. While we're at it, we may as well make the Manifest well-formed by breaking it into 72-byte > lines. See "Line length" under: > https://docs.oracle.com/en/java/javase/15/docs/specs/jar/jar.html#notes-on-manifest-and-signature-files Just write > enough lines to exceed Integer.MAX_VALUE bytes. I decided to slightly change the way this large manifest file was being created. I borrowed the idea from `Zip64SizeTest`[1] to create the file and set its length to a large value. I hope that is OK. If not, let me know, I will change this part. [1] https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/Zip64SizeTest.java#L121 ------------- PR: https://git.openjdk.java.net/jdk/pull/323 From weijun at openjdk.java.net Thu Oct 1 18:58:10 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 1 Oct 2020 18:58:10 GMT Subject: Integrated: 8249783: Simplify DerValue and DerInputStream In-Reply-To: References: Message-ID: <3ky-3noOOSDEhn60SRL7SnQQNDgITm5hxT25yiN8VxY=.8e50e255-4880-44ba-ba37-8411bc2984a4@github.com> On Thu, 17 Sep 2020 23:00:54 GMT, Weijun Wang wrote: > This code change rewrites DerValue into a mostly immutable class and simplifies DerInputStream as a wrapper for a > series of DerValues objects. DerInputBuffer is removed. > All existing methods of DerValue and DerInputStream should still work with the exact same behavior, except for a few > places where bugs are fixed. For example, Indefinite length must be used with a constructed tag. > Except for the ObjectIdentifier class where DerInputBuffer is directly referenced, no other code is touched. This pull request has now been integrated. Changeset: 3c4e824a Author: Weijun Wang URL: https://git.openjdk.java.net/jdk/commit/3c4e824a Stats: 2083 lines in 8 files changed: 616 ins; 1250 del; 217 mod 8249783: Simplify DerValue and DerInputStream Reviewed-by: valeriep ------------- PR: https://git.openjdk.java.net/jdk/pull/232 From weijun at openjdk.java.net Thu Oct 1 20:09:16 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 1 Oct 2020 20:09:16 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms Message-ID: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at https://bugs.openjdk.java.net/browse/JDK-8228481. ------------- Commit messages: - 8153005: Upgrade the default PKCS12 encryption/MAC algorithms Changes: https://git.openjdk.java.net/jdk/pull/473/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=473&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8153005 Stats: 445 lines in 6 files changed: 170 ins; 103 del; 172 mod Patch: https://git.openjdk.java.net/jdk/pull/473.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/473/head:pull/473 PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Thu Oct 1 20:09:16 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 1 Oct 2020 20:09:16 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: On Thu, 1 Oct 2020 20:02:34 GMT, Weijun Wang wrote: > Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at > https://bugs.openjdk.java.net/browse/JDK-8228481. TBD: We bumped iteration counts for PBE and HMAC to 50000 and 100000 when we were using weak algorithms. Now that the algorithms are strong, we can consider lower them. Currently, openssl 3.0.0 uses 2048 and Windows Server 2019 uses 2000. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From rriggs at openjdk.java.net Fri Oct 2 15:24:51 2020 From: rriggs at openjdk.java.net (Roger Riggs) Date: Fri, 2 Oct 2020 15:24:51 GMT Subject: RFR: 8251989: Hex formatting and parsing utility Message-ID: java.util.HexFormat utility: - Format and parse hexadecimal strings, with parameters for delimiter, prefix, suffix and upper/lowercase - Static factories and builder methods to create HexFormat copies with modified parameters. - Consistent naming of methods for conversion of byte arrays to formatted strings and back: formatHex and parseHex - Consistent naming of methods for conversion of primitive types: toHexDigits... and fromHexDigits... - Prefix and suffixes now apply to each formatted value, not the string as a whole - Using java.util.Appendable as a target for buffered conversions so output to Writers and PrintStreams like System.out are supported in addition to StringBuilder. (IOExceptions are converted to unchecked exceptions) - Immutable and thread safe, a "value-based" class See the [HexFormat javadoc](http://cr.openjdk.java.net/~rriggs/8251989-hex-formatter/java.base/java/util/HexFormat.html) for details. Review comments and suggestions welcome. ------------- Commit messages: - 8251989: Hex formatting and parsing utility Changes: https://git.openjdk.java.net/jdk/pull/482/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=482&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8251989 Stats: 1665 lines in 12 files changed: 1503 ins; 144 del; 18 mod Patch: https://git.openjdk.java.net/jdk/pull/482.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/482/head:pull/482 PR: https://git.openjdk.java.net/jdk/pull/482 From rhalade at openjdk.java.net Fri Oct 2 16:26:45 2020 From: rhalade at openjdk.java.net (Rajan Halade) Date: Fri, 2 Oct 2020 16:26:45 GMT Subject: RFR: 8239105: Add exception for expiring Digicert root certificates to =?UTF-8?B?VmVy4oCm?= Message-ID: 8239105: Add exception for expiring Digicert root certificates to Ver? ------------- Commit messages: - 8239105: Add exception for expiring Digicert root certificates to VerifyCACerts test Changes: https://git.openjdk.java.net/jdk/pull/484/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=484&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8239105 Stats: 4 lines in 1 file changed: 4 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/484.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/484/head:pull/484 PR: https://git.openjdk.java.net/jdk/pull/484 From mullan at openjdk.java.net Fri Oct 2 17:12:37 2020 From: mullan at openjdk.java.net (Sean Mullan) Date: Fri, 2 Oct 2020 17:12:37 GMT Subject: RFR: 8239105 : Add exception for expiring Digicert root certificates to VerifyCACerts test In-Reply-To: References: Message-ID: <7Ox4NjuiOLZVcJkgBs8oYj1HSgnzJ5MV1fZMSNzlObE=.3ee80dd4-df00-44c4-8029-b14a0956c14b@github.com> On Fri, 2 Oct 2020 16:21:47 GMT, Rajan Halade wrote: > 8239105 : Add exception for expiring Digicert root certificates to VerifyCACerts test Looks good. Shouldn't we be seeing mach5 test failures from this though? I didn't see any failures. ------------- Marked as reviewed by mullan (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/484 From rhalade at openjdk.java.net Fri Oct 2 17:24:39 2020 From: rhalade at openjdk.java.net (Rajan Halade) Date: Fri, 2 Oct 2020 17:24:39 GMT Subject: RFR: 8239105 : Add exception for expiring Digicert root certificates to VerifyCACerts test In-Reply-To: <7Ox4NjuiOLZVcJkgBs8oYj1HSgnzJ5MV1fZMSNzlObE=.3ee80dd4-df00-44c4-8029-b14a0956c14b@github.com> References: <7Ox4NjuiOLZVcJkgBs8oYj1HSgnzJ5MV1fZMSNzlObE=.3ee80dd4-df00-44c4-8029-b14a0956c14b@github.com> Message-ID: On Fri, 2 Oct 2020 17:09:37 GMT, Sean Mullan wrote: > Looks good. Shouldn't we be seeing mach5 test failures from this though? I didn't see any failures. We are still not beyond 90 days. It will start failing over the weekend. ------------- PR: https://git.openjdk.java.net/jdk/pull/484 From rhalade at openjdk.java.net Fri Oct 2 17:24:39 2020 From: rhalade at openjdk.java.net (Rajan Halade) Date: Fri, 2 Oct 2020 17:24:39 GMT Subject: Integrated: 8239105 : Add exception for expiring Digicert root certificates to VerifyCACerts test In-Reply-To: References: Message-ID: <9P225IYUcJI0_VHyJR80QZ1QfAbIn4HVRqfvRvOkU68=.232c0102-1aa9-4d8c-9d26-f6ae13b8f654@github.com> On Fri, 2 Oct 2020 16:21:47 GMT, Rajan Halade wrote: > 8239105 : Add exception for expiring Digicert root certificates to VerifyCACerts test This pull request has now been integrated. Changeset: 123e786d Author: Rajan Halade URL: https://git.openjdk.java.net/jdk/commit/123e786d Stats: 4 lines in 1 file changed: 4 ins; 0 del; 0 mod 8239105: Add exception for expiring Digicert root certificates to VerifyCACerts test "8239105: added verisigntsaca and thawtepremiumserverca to EXPIRY_EXC_ENTRIES list" Reviewed-by: mullan ------------- PR: https://git.openjdk.java.net/jdk/pull/484 From mullan at openjdk.java.net Fri Oct 2 18:47:40 2020 From: mullan at openjdk.java.net (Sean Mullan) Date: Fri, 2 Oct 2020 18:47:40 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: <0ajoYUlBRg1raLBhA0k4LVHy1UKw4R0hXLj4eoKByas=.e2bc71ac-14ba-480c-ba0c-177b29a1eb2c@github.com> On Thu, 1 Oct 2020 20:02:34 GMT, Weijun Wang wrote: > Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at > https://bugs.openjdk.java.net/browse/JDK-8228481. test/jdk/sun/security/mscapi/VeryLongAlias.java line 51: > 49: public static void main(String[] args) throws Throwable { > 50: > 51: // Using the old algorithms to make sure the file is recognized Do we also want to have a test that uses the new algorithms? ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From mullan at openjdk.java.net Fri Oct 2 18:51:37 2020 From: mullan at openjdk.java.net (Sean Mullan) Date: Fri, 2 Oct 2020 18:51:37 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: On Thu, 1 Oct 2020 20:02:34 GMT, Weijun Wang wrote: > Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at > https://bugs.openjdk.java.net/browse/JDK-8228481. test/lib/jdk/test/lib/security/DerUtils.java line 1: > 1: /* Is this test change supposed to be a part of this fix? ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Fri Oct 2 19:11:39 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 2 Oct 2020 19:11:39 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <0ajoYUlBRg1raLBhA0k4LVHy1UKw4R0hXLj4eoKByas=.e2bc71ac-14ba-480c-ba0c-177b29a1eb2c@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <0ajoYUlBRg1raLBhA0k4LVHy1UKw4R0hXLj4eoKByas=.e2bc71ac-14ba-480c-ba0c-177b29a1eb2c@github.com> Message-ID: On Fri, 2 Oct 2020 18:44:48 GMT, Sean Mullan wrote: >> Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at >> https://bugs.openjdk.java.net/browse/JDK-8228481. > > test/jdk/sun/security/mscapi/VeryLongAlias.java line 51: > >> 49: public static void main(String[] args) throws Throwable { >> 50: >> 51: // Using the old algorithms to make sure the file is recognized > > Do we also want to have a test that uses the new algorithms? I only know Windows Server 2019 can accept the new algorithms. > test/lib/jdk/test/lib/security/DerUtils.java line 1: > >> 1: /* > > Is this test change supposed to be a part of this fix? Yes, the change simplifies `checkAlg` calls so they don't need to convert `KnownOIDs` or `String` to `ObjectIdentifier` first. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From smarks at openjdk.java.net Fri Oct 2 19:41:38 2020 From: smarks at openjdk.java.net (Stuart Marks) Date: Fri, 2 Oct 2020 19:41:38 GMT Subject: RFR: 8156071: List.of: reduce array copying during creation In-Reply-To: References: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Message-ID: On Thu, 1 Oct 2020 06:26:39 GMT, Martin Grigorov wrote: >> Plumb new internal static factory method to trust the array passed in, avoiding unnecessary copying. JMH results for >> the benchmark show about 15% improvement for the cases that were optimized, namely the 3 to 10 fixed arg cases. >> # VM options: -verbose:gc -XX:+UseParallelGC -Xms4g -Xmx4g --enable-preview -verbose:gc -XX:+UsePara >> llelGC -Xms4g -Xmx4g -Xint >> # Warmup: 5 iterations, 1 s each >> # Measurement: 5 iterations, 2 s each >> >> WITHOUT varargs optimization: >> >> Benchmark Mode Cnt Score Error Units >> ListArgs.list00 thrpt 15 6019.539 ? 144.040 ops/ms >> ListArgs.list01 thrpt 15 1985.009 ? 40.606 ops/ms >> ListArgs.list02 thrpt 15 1854.812 ? 17.488 ops/ms >> ListArgs.list03 thrpt 15 963.866 ? 10.262 ops/ms >> ListArgs.list04 thrpt 15 908.116 ? 6.278 ops/ms >> ListArgs.list05 thrpt 15 848.607 ? 16.701 ops/ms >> ListArgs.list06 thrpt 15 822.282 ? 8.905 ops/ms >> ListArgs.list07 thrpt 15 780.057 ? 11.214 ops/ms >> ListArgs.list08 thrpt 15 745.295 ? 19.204 ops/ms >> ListArgs.list09 thrpt 15 704.596 ? 14.003 ops/ms >> ListArgs.list10 thrpt 15 696.436 ? 4.914 ops/ms >> ListArgs.list11 thrpt 15 661.908 ? 11.041 ops/ms >> >> WITH varargs optimization: >> >> Benchmark Mode Cnt Score Error Units >> ListArgs.list00 thrpt 15 6172.298 ? 62.736 ops/ms >> ListArgs.list01 thrpt 15 1987.724 ? 45.468 ops/ms >> ListArgs.list02 thrpt 15 1843.419 ? 10.693 ops/ms >> ListArgs.list03 thrpt 15 1126.946 ? 30.952 ops/ms >> ListArgs.list04 thrpt 15 1050.440 ? 17.859 ops/ms >> ListArgs.list05 thrpt 15 999.275 ? 23.656 ops/ms >> ListArgs.list06 thrpt 15 948.844 ? 19.615 ops/ms >> ListArgs.list07 thrpt 15 897.541 ? 15.531 ops/ms >> ListArgs.list08 thrpt 15 853.359 ? 18.755 ops/ms >> ListArgs.list09 thrpt 15 826.394 ? 8.284 ops/ms >> ListArgs.list10 thrpt 15 779.231 ? 4.104 ops/ms >> ListArgs.list11 thrpt 15 650.888 ? 3.948 ops/ms > > src/java.base/share/classes/jdk/internal/access/SharedSecrets.java line 88: > >> 86: if (javaUtilCollectionAccess == null) { >> 87: try { >> 88: Class.forName("java.util.ImmutableCollections$Access", true, null); > > How does this work ? It attempts to load this class but `javaUtilCollectionAccess` is never assigned to a new value. > > **Update**: I just noticed this is the getter. Yeah the SharedSecrets stuff is pretty twisted ------------- PR: https://git.openjdk.java.net/jdk/pull/449 From smarks at openjdk.java.net Fri Oct 2 19:45:36 2020 From: smarks at openjdk.java.net (Stuart Marks) Date: Fri, 2 Oct 2020 19:45:36 GMT Subject: RFR: 8156071: List.of: reduce array copying during creation In-Reply-To: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> References: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Message-ID: On Thu, 1 Oct 2020 00:13:28 GMT, Stuart Marks wrote: > Plumb new internal static factory method to trust the array passed in, avoiding unnecessary copying. JMH results for > the benchmark show about 15% improvement for the cases that were optimized, namely the 3 to 10 fixed arg cases. > # VM options: -verbose:gc -XX:+UseParallelGC -Xms4g -Xmx4g --enable-preview -verbose:gc -XX:+UsePara > llelGC -Xms4g -Xmx4g -Xint > # Warmup: 5 iterations, 1 s each > # Measurement: 5 iterations, 2 s each > > WITHOUT varargs optimization: > > Benchmark Mode Cnt Score Error Units > ListArgs.list00 thrpt 15 6019.539 ? 144.040 ops/ms > ListArgs.list01 thrpt 15 1985.009 ? 40.606 ops/ms > ListArgs.list02 thrpt 15 1854.812 ? 17.488 ops/ms > ListArgs.list03 thrpt 15 963.866 ? 10.262 ops/ms > ListArgs.list04 thrpt 15 908.116 ? 6.278 ops/ms > ListArgs.list05 thrpt 15 848.607 ? 16.701 ops/ms > ListArgs.list06 thrpt 15 822.282 ? 8.905 ops/ms > ListArgs.list07 thrpt 15 780.057 ? 11.214 ops/ms > ListArgs.list08 thrpt 15 745.295 ? 19.204 ops/ms > ListArgs.list09 thrpt 15 704.596 ? 14.003 ops/ms > ListArgs.list10 thrpt 15 696.436 ? 4.914 ops/ms > ListArgs.list11 thrpt 15 661.908 ? 11.041 ops/ms > > WITH varargs optimization: > > Benchmark Mode Cnt Score Error Units > ListArgs.list00 thrpt 15 6172.298 ? 62.736 ops/ms > ListArgs.list01 thrpt 15 1987.724 ? 45.468 ops/ms > ListArgs.list02 thrpt 15 1843.419 ? 10.693 ops/ms > ListArgs.list03 thrpt 15 1126.946 ? 30.952 ops/ms > ListArgs.list04 thrpt 15 1050.440 ? 17.859 ops/ms > ListArgs.list05 thrpt 15 999.275 ? 23.656 ops/ms > ListArgs.list06 thrpt 15 948.844 ? 19.615 ops/ms > ListArgs.list07 thrpt 15 897.541 ? 15.531 ops/ms > ListArgs.list08 thrpt 15 853.359 ? 18.755 ops/ms > ListArgs.list09 thrpt 15 826.394 ? 8.284 ops/ms > ListArgs.list10 thrpt 15 779.231 ? 4.104 ops/ms > ListArgs.list11 thrpt 15 650.888 ? 3.948 ops/ms After a hint from @cl4es I ran the benchmarks with `-prof gc`. The allocation rate is reduced by about 40% per operation in the cases where the optimization was applied. WITHOUT varargs optimization: ListArgs.list00:?gc.alloc.rate.norm thrpt 5 ? 10?? B/op ListArgs.list01:?gc.alloc.rate.norm thrpt 5 24.000 ? 0.001 B/op ListArgs.list02:?gc.alloc.rate.norm thrpt 5 24.000 ? 0.001 B/op ListArgs.list03:?gc.alloc.rate.norm thrpt 5 80.000 ? 0.001 B/op ListArgs.list04:?gc.alloc.rate.norm thrpt 5 80.036 ? 0.309 B/op ListArgs.list05:?gc.alloc.rate.norm thrpt 5 96.037 ? 0.316 B/op ListArgs.list06:?gc.alloc.rate.norm thrpt 5 96.038 ? 0.326 B/op ListArgs.list07:?gc.alloc.rate.norm thrpt 5 112.042 ? 0.361 B/op ListArgs.list08:?gc.alloc.rate.norm thrpt 5 112.043 ? 0.367 B/op ListArgs.list09:?gc.alloc.rate.norm thrpt 5 128.045 ? 0.385 B/op ListArgs.list10:?gc.alloc.rate.norm thrpt 5 128.046 ? 0.391 B/op ListArgs.list11:?gc.alloc.rate.norm thrpt 5 144.047 ? 0.406 B/op WITH varargs optimization: ListArgs.list00:?gc.alloc.rate.norm thrpt 5 ? 10?? B/op ListArgs.list01:?gc.alloc.rate.norm thrpt 5 24.000 ? 0.001 B/op ListArgs.list02:?gc.alloc.rate.norm thrpt 5 24.000 ? 0.001 B/op ListArgs.list03:?gc.alloc.rate.norm thrpt 5 48.000 ? 0.001 B/op ListArgs.list04:?gc.alloc.rate.norm thrpt 5 48.000 ? 0.001 B/op ListArgs.list05:?gc.alloc.rate.norm thrpt 5 56.000 ? 0.001 B/op ListArgs.list06:?gc.alloc.rate.norm thrpt 5 56.000 ? 0.001 B/op ListArgs.list07:?gc.alloc.rate.norm thrpt 5 64.000 ? 0.001 B/op ListArgs.list08:?gc.alloc.rate.norm thrpt 5 64.000 ? 0.001 B/op ListArgs.list09:?gc.alloc.rate.norm thrpt 5 72.000 ? 0.001 B/op ListArgs.list10:?gc.alloc.rate.norm thrpt 5 72.000 ? 0.001 B/op ListArgs.list11:?gc.alloc.rate.norm thrpt 5 144.050 ? 0.427 B/op ------------- PR: https://git.openjdk.java.net/jdk/pull/449 From psandoz at openjdk.java.net Fri Oct 2 20:41:45 2020 From: psandoz at openjdk.java.net (Paul Sandoz) Date: Fri, 2 Oct 2020 20:41:45 GMT Subject: RFR: 8156071: List.of: reduce array copying during creation In-Reply-To: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> References: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Message-ID: On Thu, 1 Oct 2020 00:13:28 GMT, Stuart Marks wrote: > Plumb new internal static factory method to trust the array passed in, avoiding unnecessary copying. JMH results for > the benchmark show about 15% improvement for the cases that were optimized, namely the 3 to 10 fixed arg cases. > # VM options: -verbose:gc -XX:+UseParallelGC -Xms4g -Xmx4g --enable-preview -verbose:gc -XX:+UsePara > llelGC -Xms4g -Xmx4g -Xint > # Warmup: 5 iterations, 1 s each > # Measurement: 5 iterations, 2 s each > > WITHOUT varargs optimization: > > Benchmark Mode Cnt Score Error Units > ListArgs.list00 thrpt 15 6019.539 ? 144.040 ops/ms > ListArgs.list01 thrpt 15 1985.009 ? 40.606 ops/ms > ListArgs.list02 thrpt 15 1854.812 ? 17.488 ops/ms > ListArgs.list03 thrpt 15 963.866 ? 10.262 ops/ms > ListArgs.list04 thrpt 15 908.116 ? 6.278 ops/ms > ListArgs.list05 thrpt 15 848.607 ? 16.701 ops/ms > ListArgs.list06 thrpt 15 822.282 ? 8.905 ops/ms > ListArgs.list07 thrpt 15 780.057 ? 11.214 ops/ms > ListArgs.list08 thrpt 15 745.295 ? 19.204 ops/ms > ListArgs.list09 thrpt 15 704.596 ? 14.003 ops/ms > ListArgs.list10 thrpt 15 696.436 ? 4.914 ops/ms > ListArgs.list11 thrpt 15 661.908 ? 11.041 ops/ms > > WITH varargs optimization: > > Benchmark Mode Cnt Score Error Units > ListArgs.list00 thrpt 15 6172.298 ? 62.736 ops/ms > ListArgs.list01 thrpt 15 1987.724 ? 45.468 ops/ms > ListArgs.list02 thrpt 15 1843.419 ? 10.693 ops/ms > ListArgs.list03 thrpt 15 1126.946 ? 30.952 ops/ms > ListArgs.list04 thrpt 15 1050.440 ? 17.859 ops/ms > ListArgs.list05 thrpt 15 999.275 ? 23.656 ops/ms > ListArgs.list06 thrpt 15 948.844 ? 19.615 ops/ms > ListArgs.list07 thrpt 15 897.541 ? 15.531 ops/ms > ListArgs.list08 thrpt 15 853.359 ? 18.755 ops/ms > ListArgs.list09 thrpt 15 826.394 ? 8.284 ops/ms > ListArgs.list10 thrpt 15 779.231 ? 4.104 ops/ms > ListArgs.list11 thrpt 15 650.888 ? 3.948 ops/ms Looks good, i wondered why the performance results were so slow then i looked more closely and saw "-Xint" was used. I usually don't ascribe much value to micro benchmarks run in interpreter only mode, but hey any shaving off startup time is welcome. Less allocation is definitely welcome (although i do wish C2 was better at eliding redundant array initialization and allocation). ------------- Marked as reviewed by psandoz (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/449 From alanb at openjdk.java.net Sun Oct 4 08:44:38 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Sun, 4 Oct 2020 08:44:38 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA In-Reply-To: References: Message-ID: On Wed, 23 Sep 2020 14:41:59 GMT, Weijun Wang wrote: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Changes requested by alanb (Reviewer). test/lib/jdk/test/lib/util/JarUtils.java line 90: > 88: String name = toJarEntryName(entry); > 89: jos.putNextEntry(new JarEntry(name)); > 90: if (Files.exists(dir.resolve(entry))) { This is test infrastructure that we use in several areas and changing it to allow file paths to files that don't exist be problematic. Is there any reason why the jarsigner can't create an empty or dummy file to put into the JAR file? ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Sun Oct 4 13:34:38 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Sun, 4 Oct 2020 13:34:38 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA In-Reply-To: References: Message-ID: On Sun, 4 Oct 2020 08:41:28 GMT, Alan Bateman wrote: >> Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: >> >> - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner >> >> - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature >> algorithms >> >> - A new JarSigner property "directsign" >> >> - Updating the jarsigner tool doc >> >> Major code changes: >> >> - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm >> there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. >> >> - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java >> >> - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId >> >> - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing >> >> - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms >> >> - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed > > test/lib/jdk/test/lib/util/JarUtils.java line 90: > >> 88: String name = toJarEntryName(entry); >> 89: jos.putNextEntry(new JarEntry(name)); >> 90: if (Files.exists(dir.resolve(entry))) { > > This is test infrastructure that we use in several areas and changing it to allow file paths to files that don't exist > be problematic. Is there any reason why the jarsigner can't create an empty or dummy file to put into the JAR file? Sorry, I'll revert the change and create files myself. I just thought any existing call to this method should have the file already created there, but it could be a problem if the creation is not trivial and might fail. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Sun Oct 4 14:02:49 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Sun, 4 Oct 2020 14:02:49 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v2] In-Reply-To: References: Message-ID: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Weijun Wang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits: - extract some methods, rollback JarUtils, deperated version from 15 to 16. - Merge - 8242068: Signed JAR support for RSASSA-PSS and EdDSA ------------- Changes: https://git.openjdk.java.net/jdk/pull/322/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=01 Stats: 1696 lines in 21 files changed: 972 ins; 555 del; 169 mod Patch: https://git.openjdk.java.net/jdk/pull/322.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/322/head:pull/322 PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Sun Oct 4 14:09:48 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Sun, 4 Oct 2020 14:09:48 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v3] In-Reply-To: References: Message-ID: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Weijun Wang has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: extract some methods, rollback JarUtils, deprecated version from 15 to 16. ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/322/files - new: https://git.openjdk.java.net/jdk/pull/322/files/c05af70a..5d455bae Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=01-02 Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/322.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/322/head:pull/322 PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Sun Oct 4 14:12:37 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Sun, 4 Oct 2020 14:12:37 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v3] In-Reply-To: References: Message-ID: On Sun, 4 Oct 2020 08:41:40 GMT, Alan Bateman wrote: >> Weijun Wang has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. > > Changes requested by alanb (Reviewer). Note: I force pushed a new commit to correct a typo in the summary line. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Sun Oct 4 17:29:50 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Sun, 4 Oct 2020 17:29:50 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v4] In-Reply-To: References: Message-ID: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: fotgot to read one property ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/322/files - new: https://git.openjdk.java.net/jdk/pull/322/files/5d455bae..fecf30d7 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=02-03 Stats: 2 lines in 1 file changed: 2 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/322.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/322/head:pull/322 PR: https://git.openjdk.java.net/jdk/pull/322 From smarks at openjdk.java.net Mon Oct 5 17:03:48 2020 From: smarks at openjdk.java.net (Stuart Marks) Date: Mon, 5 Oct 2020 17:03:48 GMT Subject: Integrated: 8156071: List.of: reduce array copying during creation In-Reply-To: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> References: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Message-ID: On Thu, 1 Oct 2020 00:13:28 GMT, Stuart Marks wrote: > Plumb new internal static factory method to trust the array passed in, avoiding unnecessary copying. JMH results for > the benchmark show about 15% improvement for the cases that were optimized, namely the 3 to 10 fixed arg cases. > # VM options: -verbose:gc -XX:+UseParallelGC -Xms4g -Xmx4g --enable-preview -verbose:gc -XX:+UsePara > llelGC -Xms4g -Xmx4g -Xint > # Warmup: 5 iterations, 1 s each > # Measurement: 5 iterations, 2 s each > > WITHOUT varargs optimization: > > Benchmark Mode Cnt Score Error Units > ListArgs.list00 thrpt 15 6019.539 ? 144.040 ops/ms > ListArgs.list01 thrpt 15 1985.009 ? 40.606 ops/ms > ListArgs.list02 thrpt 15 1854.812 ? 17.488 ops/ms > ListArgs.list03 thrpt 15 963.866 ? 10.262 ops/ms > ListArgs.list04 thrpt 15 908.116 ? 6.278 ops/ms > ListArgs.list05 thrpt 15 848.607 ? 16.701 ops/ms > ListArgs.list06 thrpt 15 822.282 ? 8.905 ops/ms > ListArgs.list07 thrpt 15 780.057 ? 11.214 ops/ms > ListArgs.list08 thrpt 15 745.295 ? 19.204 ops/ms > ListArgs.list09 thrpt 15 704.596 ? 14.003 ops/ms > ListArgs.list10 thrpt 15 696.436 ? 4.914 ops/ms > ListArgs.list11 thrpt 15 661.908 ? 11.041 ops/ms > > WITH varargs optimization: > > Benchmark Mode Cnt Score Error Units > ListArgs.list00 thrpt 15 6172.298 ? 62.736 ops/ms > ListArgs.list01 thrpt 15 1987.724 ? 45.468 ops/ms > ListArgs.list02 thrpt 15 1843.419 ? 10.693 ops/ms > ListArgs.list03 thrpt 15 1126.946 ? 30.952 ops/ms > ListArgs.list04 thrpt 15 1050.440 ? 17.859 ops/ms > ListArgs.list05 thrpt 15 999.275 ? 23.656 ops/ms > ListArgs.list06 thrpt 15 948.844 ? 19.615 ops/ms > ListArgs.list07 thrpt 15 897.541 ? 15.531 ops/ms > ListArgs.list08 thrpt 15 853.359 ? 18.755 ops/ms > ListArgs.list09 thrpt 15 826.394 ? 8.284 ops/ms > ListArgs.list10 thrpt 15 779.231 ? 4.104 ops/ms > ListArgs.list11 thrpt 15 650.888 ? 3.948 ops/ms This pull request has now been integrated. Changeset: 88d75c9a Author: Stuart Marks URL: https://git.openjdk.java.net/jdk/commit/88d75c9a Stats: 203 lines in 6 files changed: 185 ins; 0 del; 18 mod 8156071: List.of: reduce array copying during creation Reviewed-by: psandoz, redestad ------------- PR: https://git.openjdk.java.net/jdk/pull/449 From tvaleev at openjdk.java.net Tue Oct 6 03:14:41 2020 From: tvaleev at openjdk.java.net (Tagir F.Valeev) Date: Tue, 6 Oct 2020 03:14:41 GMT Subject: RFR: 8156071: List.of: reduce array copying during creation In-Reply-To: References: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Message-ID: On Fri, 2 Oct 2020 20:38:40 GMT, Paul Sandoz wrote: >> Plumb new internal static factory method to trust the array passed in, avoiding unnecessary copying. JMH results for >> the benchmark show about 15% improvement for the cases that were optimized, namely the 3 to 10 fixed arg cases. >> # VM options: -verbose:gc -XX:+UseParallelGC -Xms4g -Xmx4g --enable-preview -verbose:gc -XX:+UsePara >> llelGC -Xms4g -Xmx4g -Xint >> # Warmup: 5 iterations, 1 s each >> # Measurement: 5 iterations, 2 s each >> >> WITHOUT varargs optimization: >> >> Benchmark Mode Cnt Score Error Units >> ListArgs.list00 thrpt 15 6019.539 ? 144.040 ops/ms >> ListArgs.list01 thrpt 15 1985.009 ? 40.606 ops/ms >> ListArgs.list02 thrpt 15 1854.812 ? 17.488 ops/ms >> ListArgs.list03 thrpt 15 963.866 ? 10.262 ops/ms >> ListArgs.list04 thrpt 15 908.116 ? 6.278 ops/ms >> ListArgs.list05 thrpt 15 848.607 ? 16.701 ops/ms >> ListArgs.list06 thrpt 15 822.282 ? 8.905 ops/ms >> ListArgs.list07 thrpt 15 780.057 ? 11.214 ops/ms >> ListArgs.list08 thrpt 15 745.295 ? 19.204 ops/ms >> ListArgs.list09 thrpt 15 704.596 ? 14.003 ops/ms >> ListArgs.list10 thrpt 15 696.436 ? 4.914 ops/ms >> ListArgs.list11 thrpt 15 661.908 ? 11.041 ops/ms >> >> WITH varargs optimization: >> >> Benchmark Mode Cnt Score Error Units >> ListArgs.list00 thrpt 15 6172.298 ? 62.736 ops/ms >> ListArgs.list01 thrpt 15 1987.724 ? 45.468 ops/ms >> ListArgs.list02 thrpt 15 1843.419 ? 10.693 ops/ms >> ListArgs.list03 thrpt 15 1126.946 ? 30.952 ops/ms >> ListArgs.list04 thrpt 15 1050.440 ? 17.859 ops/ms >> ListArgs.list05 thrpt 15 999.275 ? 23.656 ops/ms >> ListArgs.list06 thrpt 15 948.844 ? 19.615 ops/ms >> ListArgs.list07 thrpt 15 897.541 ? 15.531 ops/ms >> ListArgs.list08 thrpt 15 853.359 ? 18.755 ops/ms >> ListArgs.list09 thrpt 15 826.394 ? 8.284 ops/ms >> ListArgs.list10 thrpt 15 779.231 ? 4.104 ops/ms >> ListArgs.list11 thrpt 15 650.888 ? 3.948 ops/ms > > Looks good, i wondered why the performance results were so slow then i looked more closely and saw "-Xint" was used. I > usually don't ascribe much value to micro benchmarks run in interpreter only mode, but hey any shaving off startup time > is welcome. Less allocation is definitely welcome (although i do wish C2 was better at eliding redundant array > initialization and allocation). Sorry to be late to the party. I thought that all reviews labeled with core-libs should be mirrored to core-libs-dev mailing list but I haven't seen it there :( Please note that the integrated implementation exposes listFromTrustedArray to everybody. No dirty unsafe reflection is necessary, only single unchecked cast: static List untrustedArrayToList(T[] array) { @SuppressWarnings("unchecked") Function, List> finisher = (Function, List>) Collectors.toUnmodifiableList().finisher(); ArrayList list = new ArrayList<>() { @Override public Object[] toArray() { return array; } }; return finisher.apply(list); } This might be qualified as a security issue. ------------- PR: https://git.openjdk.java.net/jdk/pull/449 From ascarpino at openjdk.java.net Tue Oct 6 05:03:46 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Tue, 6 Oct 2020 05:03:46 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM Message-ID: 8253821: Improve ByteBuffer performance with GCM ------------- Commit messages: - comments, dead code, change maxlen tp 1k - remove debug line - removing comments and unused method - Moved into Git Changes: https://git.openjdk.java.net/jdk/pull/411/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=411&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8253821 Stats: 842 lines in 11 files changed: 739 ins; 29 del; 74 mod Patch: https://git.openjdk.java.net/jdk/pull/411.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/411/head:pull/411 PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Tue Oct 6 05:03:46 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Tue, 6 Oct 2020 05:03:46 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM In-Reply-To: References: Message-ID: <12eT2pJ_ZEgVVAbxtG5GgaurGiCxxILuvMS1yYaRLn4=.1ae7c11f-b9de-43fc-82f7-1c23d93c6bad@github.com> On Tue, 29 Sep 2020 20:22:55 GMT, Anthony Scarpino wrote: > 8253821: Improve ByteBuffer performance with GCM I'd like a review of this change. They are two performance improvements to AES-GCM, the larger being the usage with ByteBuffers. Below are the details of the change and are listed in the JBS bug description, any future comments will be applied to the bug: There were two areas of focus, the primary is when direct bytebuffers are used with some crypto algorithms, data is copied to byte arrays numerous times, causing unnecessary memory allocation and bringing down performance. The other focus was the non-direct bytebuffer output arrays. This change comes in multiple parts: 1) Changing CipherCore to not allocate a new output array if the existing array is large enough. Create a new array only if the length is not enough. The only SunJCE algorithm that has special output needs is GCM which can be dealt with elsewhere. 2) AESCipher has a one-size-fits-all approach to bytebuffers. All encryption and decryption is done in byte arrays. When the input data is a byte array or a bytebuffer backed by a byte array, this is ok. However when it is a direct buffer, the data is copied into a new byte array. Unfortunately, this hurts SSLEngine which uses direct buffers causing multiple copies of data down to the raw algorithm. Additionally GCM code and other related classes had to be changed to allow ByteBuffers down to the algorithm where it can be copied into a fixed size byte array that can be reused. Code without this modifications running JFR with Flink, a performance test, shows ~150GB of byte array allocation in one minute of operation, afterward 7GB. 3) GCM needed some reworking of the logic. Being an authenticated cipher, if the GHASH check fails, the decryption fails and no data is returned. The existing code would perform the decryption at the same time as the GHASH check, which current design offers no parallel performance advantage. Performing GHASH fully before decryption prevents allocating output data and perform unneeded operations if the GHASH is failed. If GHASH is successful, in-place operations can be performed directly to the buffer without allocating an intermediary buffer and then copying that data. 4) GCTR and GHASH allocating a fixed buffer size if the data size is over 1k when going into an intrinsic. At this time copying data from the bytebuffer into a byte array for the intrinsic to work on it is required. We cannot eliminate the copy, but we can reduce the size of the allocated buffer. There is little harm in creating a maximum size this buffer can be and copy data into that buffer repeatedly until it is finished. Having the maximum size at 4k does produce slightly faster top-end performance at times, but inconsistent results and an increase in memory usage from 7GB to 17GB have been inconclusive to increase the buffer size. 5) Using bytebuffers allows for using duplicate() which lets the code easier chop up the data without unnecessary copying The CipherCore change provided a 6% performance gains for GCM with byte array based data, such as SSLSocket and direct API calls. Similar performance gains should be evident with other algorithms using this method. The GCM bytebuffer and logic changes produced a 16% increase in performance in the Flink test. This is limited to only GCM as the other algorithms still use bytebuffer-to-byte array copy method. Doing similar work on other algorithms would provide less of a performance gain because of the complexities of GCM and are have diminishing usage in TLS. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From tvaleev at openjdk.java.net Tue Oct 6 05:10:42 2020 From: tvaleev at openjdk.java.net (Tagir F.Valeev) Date: Tue, 6 Oct 2020 05:10:42 GMT Subject: RFR: 8156071: List.of: reduce array copying during creation In-Reply-To: References: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Message-ID: On Tue, 6 Oct 2020 03:10:34 GMT, Tagir F. Valeev wrote: >> Looks good, i wondered why the performance results were so slow then i looked more closely and saw "-Xint" was used. I >> usually don't ascribe much value to micro benchmarks run in interpreter only mode, but hey any shaving off startup time >> is welcome. Less allocation is definitely welcome (although i do wish C2 was better at eliding redundant array >> initialization and allocation). > > Sorry to be late to the party. I thought that all reviews labeled with core-libs should be mirrored to core-libs-dev > mailing list but I haven't seen it there :( > Please note that the integrated implementation exposes listFromTrustedArray to everybody. No dirty unsafe reflection is > necessary, only single unchecked cast: > static List untrustedArrayToList(T[] array) { > @SuppressWarnings("unchecked") > Function, List> finisher = > (Function, List>) Collectors.toUnmodifiableList().finisher(); > ArrayList list = new ArrayList<>() { > @Override > public Object[] toArray() { > return array; > } > }; > return finisher.apply(list); > } > > This might be qualified as a security issue. This could be fixed by adding a classword check to the finisher, like this: list -> { if (list.getClass() != ArrayList.class) { throw new IllegalArgumentException(); } return (List) SharedSecrets.getJavaUtilCollectionAccess() .listFromTrustedArray(list.toArray()); }, ------------- PR: https://git.openjdk.java.net/jdk/pull/449 From simonis at openjdk.java.net Tue Oct 6 13:51:21 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Tue, 6 Oct 2020 13:51:21 GMT Subject: RFR: 8253952: Work around wrong usage of ZipOutputStream.putNextEntry() in user code Message-ID: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> ### Summary Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid entry compressed size"`. ### Motivation In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as follows: ZipEntry entry; ZipInputStream zis = new ZipInputStream(...); ZipOutputStream zos = new ZipOutputStream(...); while((entry = zis.getNextEntry()) != null) { zos.putNextEntry(entry); zis.transferTo(zos); } The problem with this code is that the zip file format does not record the compression level used for deflation in its entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at least resetting of the compressed size field. E.g.: while((entry = zis.getNextEntry()) != null) { ZipEntry newEntry = new ZipEntry(entry.getName()); zos.putNextEntry(newEntry); zis.transferTo(zos); } or: while((entry = zis.getNextEntry()) != null) { entry.setCompressedSize(-1); zos.putNextEntry(entry); zis.transferTo(zos); } Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for selected services in production and the only reason why we haven't enabled them by default until now is the problem I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because their compression ratio is slightly different from that of the default zlib library. This can easily trigger a `ZipException` even if an application is not using a different compression levels but just a zip file created with another zlib version. I'd therefore like to propose the following workaround for the wrong `ZipOutputStream.putNextEntry()` usage in user code: - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling `ZipEntry.setCompressedSize()`. - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when reading that entry from a `ZipFile` or `ZipInputStream`. ### Technical Details A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will initially only contain the information from the LFH. Only after the next entry was read (or after `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it will give us the freedom to use whatever zip implementation we like :) [1]: https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 [2]: https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) ------------- Commit messages: - 8253952: Work around wrong usage of ZipOutputStream.putNextEntry() in user code Changes: https://git.openjdk.java.net/jdk/pull/520/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8253952 Stats: 244 lines in 4 files changed: 242 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/520.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/520/head:pull/520 PR: https://git.openjdk.java.net/jdk/pull/520 From rhalade at openjdk.java.net Tue Oct 6 16:05:13 2020 From: rhalade at openjdk.java.net (Rajan Halade) Date: Tue, 6 Oct 2020 16:05:13 GMT Subject: RFR: 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate Message-ID: 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate ------------- Commit messages: - 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate Changes: https://git.openjdk.java.net/jdk/pull/528/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=528&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8254081 Stats: 4 lines in 1 file changed: 3 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/528.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/528/head:pull/528 PR: https://git.openjdk.java.net/jdk/pull/528 From xuelei at openjdk.java.net Tue Oct 6 16:24:09 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Tue, 6 Oct 2020 16:24:09 GMT Subject: RFR: 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate In-Reply-To: References: Message-ID: On Tue, 6 Oct 2020 16:00:12 GMT, Rajan Halade wrote: > 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate test/jdk/java/security/cert/PolicyNode/GetPolicyQualifiers.java line 57: > 55: params.setRevocationEnabled(false); > 56: // Certificates expired on Oct 6th, 2020 > 57: params.setDate(new Date("July 01, 2020")); The Date() constructor is deprecated, would you like to use the replacement method DateFormat.parse()? ------------- PR: https://git.openjdk.java.net/jdk/pull/528 From rhalade at openjdk.java.net Tue Oct 6 16:33:22 2020 From: rhalade at openjdk.java.net (Rajan Halade) Date: Tue, 6 Oct 2020 16:33:22 GMT Subject: RFR: 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate [v2] In-Reply-To: References: Message-ID: > 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate Rajan Halade has updated the pull request incrementally with one additional commit since the last revision: 8254081: Use DateFormat instead of Date ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/528/files - new: https://git.openjdk.java.net/jdk/pull/528/files/f8ff3bbc..0de3b2a4 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=528&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=528&range=00-01 Stats: 5 lines in 1 file changed: 3 ins; 1 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/528.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/528/head:pull/528 PR: https://git.openjdk.java.net/jdk/pull/528 From mullan at openjdk.java.net Tue Oct 6 16:33:22 2020 From: mullan at openjdk.java.net (Sean Mullan) Date: Tue, 6 Oct 2020 16:33:22 GMT Subject: RFR: 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate [v2] In-Reply-To: References: Message-ID: On Tue, 6 Oct 2020 16:29:55 GMT, Rajan Halade wrote: >> 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate > > Rajan Halade has updated the pull request incrementally with one additional commit since the last revision: > > 8254081: Use DateFormat instead of Date Marked as reviewed by mullan (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/528 From rhalade at openjdk.java.net Tue Oct 6 16:33:23 2020 From: rhalade at openjdk.java.net (Rajan Halade) Date: Tue, 6 Oct 2020 16:33:23 GMT Subject: RFR: 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate [v2] In-Reply-To: References: Message-ID: <3aOWbzYwxNIqynf9q0g-dvCmzr3UmMHLaiN8zxqE0ak=.1f6d9f20-add3-42bc-add5-bd4c62ad5168@github.com> On Tue, 6 Oct 2020 16:21:34 GMT, Xue-Lei Andrew Fan wrote: >> Rajan Halade has updated the pull request incrementally with one additional commit since the last revision: >> >> 8254081: Use DateFormat instead of Date > > test/jdk/java/security/cert/PolicyNode/GetPolicyQualifiers.java line 57: > >> 55: params.setRevocationEnabled(false); >> 56: // Certificates expired on Oct 6th, 2020 >> 57: params.setDate(new Date("July 01, 2020")); > > The Date() constructor is deprecated, would you like to use the replacement method DateFormat.parse()? I have fixed it, thanks! ------------- PR: https://git.openjdk.java.net/jdk/pull/528 From xuelei at openjdk.java.net Tue Oct 6 16:42:10 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Tue, 6 Oct 2020 16:42:10 GMT Subject: RFR: 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate [v2] In-Reply-To: References: Message-ID: On Tue, 6 Oct 2020 16:33:22 GMT, Rajan Halade wrote: >> 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate > > Rajan Halade has updated the pull request incrementally with one additional commit since the last revision: > > 8254081: Use DateFormat instead of Date Marked as reviewed by xuelei (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/528 From rhalade at openjdk.java.net Tue Oct 6 16:42:10 2020 From: rhalade at openjdk.java.net (Rajan Halade) Date: Tue, 6 Oct 2020 16:42:10 GMT Subject: Integrated: 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate In-Reply-To: References: Message-ID: On Tue, 6 Oct 2020 16:00:12 GMT, Rajan Halade wrote: > 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate This pull request has now been integrated. Changeset: 54b340b4 Author: Rajan Halade URL: https://git.openjdk.java.net/jdk/commit/54b340b4 Stats: 6 lines in 1 file changed: 5 ins; 0 del; 1 mod 8254081: java/security/cert/PolicyNode/GetPolicyQualifiers.java fails due to an expired certificate Perform backdated validation of test certificate. Reviewed-by: mullan, xuelei ------------- PR: https://git.openjdk.java.net/jdk/pull/528 From smarks at openjdk.java.net Tue Oct 6 17:48:09 2020 From: smarks at openjdk.java.net (Stuart Marks) Date: Tue, 6 Oct 2020 17:48:09 GMT Subject: RFR: 8156071: List.of: reduce array copying during creation In-Reply-To: References: <-pkG69_yCpZxK0LBMz3Dw_m2VkTK6BhhWBY02UkidfQ=.c1f424e8-5716-4858-aaa0-488a9e553107@github.com> Message-ID: On Tue, 6 Oct 2020 05:07:37 GMT, Tagir F. Valeev wrote: >> Sorry to be late to the party. I thought that all reviews labeled with core-libs should be mirrored to core-libs-dev >> mailing list but I haven't seen it there :( >> Please note that the integrated implementation exposes listFromTrustedArray to everybody. No dirty unsafe reflection is >> necessary, only single unchecked cast: >> static List untrustedArrayToList(T[] array) { >> @SuppressWarnings("unchecked") >> Function, List> finisher = >> (Function, List>) Collectors.toUnmodifiableList().finisher(); >> ArrayList list = new ArrayList<>() { >> @Override >> public Object[] toArray() { >> return array; >> } >> }; >> return finisher.apply(list); >> } >> >> This might be qualified as a security issue. > > This could be fixed by adding a classword check to the finisher, like this: > > list -> { > if (list.getClass() != ArrayList.class) { > throw new IllegalArgumentException(); > } > return (List) SharedSecrets.getJavaUtilCollectionAccess() > .listFromTrustedArray(list.toArray()); > }, Thanks for pointing this out. I've filed bug [JDK-8254090](https://bugs.openjdk.java.net/browse/JDK-8254090). I think we're ok as long as this gets fixed before JDK 16 ships. I think the notification messages for this did end up on core-libs-dev, but perhaps there were some email delays over the weekend. ------------- PR: https://git.openjdk.java.net/jdk/pull/449 From mullan at openjdk.java.net Tue Oct 6 18:37:07 2020 From: mullan at openjdk.java.net (Sean Mullan) Date: Tue, 6 Oct 2020 18:37:07 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <0ajoYUlBRg1raLBhA0k4LVHy1UKw4R0hXLj4eoKByas=.e2bc71ac-14ba-480c-ba0c-177b29a1eb2c@github.com> Message-ID: On Fri, 2 Oct 2020 19:07:20 GMT, Weijun Wang wrote: >> test/jdk/sun/security/mscapi/VeryLongAlias.java line 51: >> >>> 49: public static void main(String[] args) throws Throwable { >>> 50: >>> 51: // Using the old algorithms to make sure the file is recognized >> >> Do we also want to have a test that uses the new algorithms? > > I only know Windows Server 2019 can accept the new algorithms. Ok, but maybe we can split this test in two and use the jtreg @requires tag to run the newer algorithms on Windows Server 2019? It would be a useful test if this is the only test where we test PKCS12 interop with Windows. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Tue Oct 6 19:04:07 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 6 Oct 2020 19:04:07 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <0ajoYUlBRg1raLBhA0k4LVHy1UKw4R0hXLj4eoKByas=.e2bc71ac-14ba-480c-ba0c-177b29a1eb2c@github.com> Message-ID: On Tue, 6 Oct 2020 18:34:34 GMT, Sean Mullan wrote: >> I only know Windows Server 2019 can accept the new algorithms. > > Ok, but maybe we can split this test in two and use the jtreg @requires tag to run the newer algorithms on Windows > Server 2019? It would be a useful test if this is the only test where we test PKCS12 interop with Windows. OK. Or I can see if there is an existing method in test/lib that can detects the version. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From lancea at openjdk.java.net Wed Oct 7 14:48:25 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Wed, 7 Oct 2020 14:48:25 GMT Subject: RFR: 8253952: Work around wrong usage of ZipOutputStream.putNextEntry() in user code In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 6 Oct 2020 10:02:09 GMT, Volker Simonis wrote: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) src/java.base/share/classes/java/util/zip/ZipOutputStream.java line 197: > 195: * Unless explictely set by calling {@link ZipEntry#setCompressedSize(long)} > 196: * this output stream will ignore the compressed size of a {@code ZipEntry} > 197: * and re-compute its value automatically after the associted data has been typo "associated" -> "associated" ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Wed Oct 7 14:56:16 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Wed, 7 Oct 2020 14:56:16 GMT Subject: RFR: 8253952: Work around wrong usage of ZipOutputStream.putNextEntry() in user code In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: <_jTazFtrOPacCLAjyCVNmkt6qZffvqOKLdasvvKYJP0=.d558846b-05ce-4444-a435-6bc9cd166a39@github.com> On Tue, 6 Oct 2020 10:02:09 GMT, Volker Simonis wrote: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) src/java.base/share/classes/java/util/zip/ZipOutputStream.java line 186: > 184: * the entry has no set modification time. > 185: * > 186: * The zip file format does not record the compression level used for the "zip" ->"ZIP" for consistency' Should this refer to the Zip File Format specification? ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From alanb at openjdk.java.net Wed Oct 7 15:13:11 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Wed, 7 Oct 2020 15:13:11 GMT Subject: RFR: 8253952: Work around wrong usage of ZipOutputStream.putNextEntry() in user code In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 6 Oct 2020 10:02:09 GMT, Volker Simonis wrote: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) src/java.base/share/classes/java/util/jar/JarOutputStream.java line 97: > 95: * and re-compute its value automatically after the associted data has been > 96: * completely deflated. > 97: * We probably need to do a bit wordsmithing here. I think I'd drop the first two sentences and instead start a new paragraph here (

tag) with "Unless explicitly set, the output stream will ignore ..." and see how that looks. Probably should be "the ZipEntry" rather than "a ZipEntry" to be consistent with the parameter description. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Wed Oct 7 15:37:06 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Wed, 7 Oct 2020 15:37:06 GMT Subject: RFR: 8253952: Work around wrong usage of ZipOutputStream.putNextEntry() in user code In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Wed, 7 Oct 2020 15:10:06 GMT, Alan Bateman wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > src/java.base/share/classes/java/util/jar/JarOutputStream.java line 97: > >> 95: * and re-compute its value automatically after the associted data has been >> 96: * completely deflated. >> 97: * > > We probably need to do a bit wordsmithing here. > > I think I'd drop the first two sentences and instead start a new paragraph here (

tag) with "Unless explicitly set, > the output stream will ignore ..." and see how that looks. > Probably should be "the ZipEntry" rather than "a ZipEntry" to be consistent with the parameter description. Alan makes a good point. Perhaps we keep things simple and focus solely on tweaking the description of the change in behavior which is described in your last paragraph ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Wed Oct 7 16:12:12 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Wed, 7 Oct 2020 16:12:12 GMT Subject: RFR: 8253952: Work around wrong usage of ZipOutputStream.putNextEntry() in user code In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 6 Oct 2020 10:02:09 GMT, Volker Simonis wrote: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) I think as we start to move forward with the review and CSR, we should update the bug and PR description as the change is not a workaround but a change in behavior of the implementation ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Wed Oct 7 16:48:13 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Wed, 7 Oct 2020 16:48:13 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to ignore ZipEntry's compressed size In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Wed, 7 Oct 2020 16:09:47 GMT, Lance Andersen wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > I think as we start to move forward with the review and CSR, we should update the bug and PR description as the change > is not a workaround but a change in behavior of the implementation I already changed the bug description on Alan's request and just updated the PR description to reflect that. Please feel free to propose a better description. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Wed Oct 7 16:48:14 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Wed, 7 Oct 2020 16:48:14 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to ignore ZipEntry's compressed size In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Wed, 7 Oct 2020 15:34:32 GMT, Lance Andersen wrote: >> src/java.base/share/classes/java/util/jar/JarOutputStream.java line 97: >> >>> 95: * and re-compute its value automatically after the associted data has been >>> 96: * completely deflated. >>> 97: * >> >> We probably need to do a bit wordsmithing here. >> >> I think I'd drop the first two sentences and instead start a new paragraph here (

tag) with "Unless explicitly set, >> the output stream will ignore ..." and see how that looks. >> Probably should be "the ZipEntry" rather than "a ZipEntry" to be consistent with the parameter description. > > Alan makes a good point. Perhaps we keep things simple and focus solely on tweaking the description of the change in > behavior which is described in your last paragraph I totally agree. What about using just the last sentence (as you've proposed) in the spec section and add the other to as @implNote? O you think the last sentence will be enough? ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Wed Oct 7 17:07:08 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Wed, 7 Oct 2020 17:07:08 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to ignore ZipEntry's compressed size In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Wed, 7 Oct 2020 16:43:53 GMT, Volker Simonis wrote: >> Alan makes a good point. Perhaps we keep things simple and focus solely on tweaking the description of the change in >> behavior which is described in your last paragraph > > I totally agree. What about using just the last sentence (as you've proposed) in the spec section and add the other to > as @implNote? O you think the last sentence will be enough? I think we can just go with the last sentence/paragraph. Perhaps we can further simplify the paragraph/sentence with something like: The compressed entry size will be recalculated for compressed (DEFLATED) entries when ZipEntry::setCompressedSize has not been explicitly called on the ZipEntry. or The compressed (DEFLATED) entry size will be recalculated when ZipEntry::setCompressedSize has not been explicitly called on the ZipEntry. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From xuelei at openjdk.java.net Wed Oct 7 17:23:11 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Wed, 7 Oct 2020 17:23:11 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM In-Reply-To: References: Message-ID: On Tue, 29 Sep 2020 20:22:55 GMT, Anthony Scarpino wrote: > 8253821: Improve ByteBuffer performance with GCM Impressive update and performance improvement! I have no major concerns, all comments are just about trivial details like indents. src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 660: > 658: } > 659: > 660: /** There is one extra indent.. src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 664: > 662: * engineUpdate() and engineDoFinal(). > 663: */ > 664: private int bufferCrypt(ByteBuffer input, ByteBuffer output, It looks like this method is copied from the CipherSpi. For maintenance, it would be nice to add an extra comment to state the copy and update. For example, "this method and implementation is copied from javax.crypto.CipherSpi, with an improvement for GCM mode." src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 947: > 945: // create temporary output buffer if the estimated size is larger > 946: // than the user-provided buffer. > 947: if (output.length - outputOffset < estOutSize) { "outputCapacity" could be used to replace "output.length - outputOffset", and join the clause with the if-clause above. src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 928: > 926: int outputCapacity = checkOutputCapacity(output, outputOffset, > 927: estOutSize); > 928: int offset = outputOffset; // 0 for decrypting the line comment, "// 0 for decrypting", could be remove as the expression get changed. src/java.base/share/classes/com/sun/crypto/provider/CounterMode.java line 28: > 26: package com.sun.crypto.provider; > 27: > 28: import java.nio.ByteBuffer; There is no more update except this line. The new import ByteBuffer may not be used. src/java.base/share/classes/com/sun/crypto/provider/FeedbackCipher.java line 250: > 248: * ByteBuffer methods should not be accessed as CipherCore and AESCipher > 249: * copy the data to byte arrays. These methods are to satisfy the compiler. > 250: * there is an extra blank comment line. src/java.base/share/classes/com/sun/crypto/provider/GHASH.java line 203: > 201: > 202: // Maximum buffer size rotating ByteBuffer->byte[] intrinsic copy > 203: static final int MAX_LEN = 1024; This filed could be private. I would like to declare class fields in the beginning of a class, for easy eyes-searching. Maybe, it is nice to use a multiple of the block size (for example 64 * AES_BLOCK_SIZE), just in case someone else update it to a weird value later. src/java.base/share/classes/com/sun/crypto/provider/GCTR.java line 144: > 142: > 143: // Maximum buffer size rotating ByteBuffer->byte[] intrinsic copy > 144: final int MAX_LEN = 1024; This filed could be private. I would like to declare class fields in the beginning of a class, for easy eyes-searching. Maybe, it is nice to use a multiple of the block size (for example 64 * AES_BLOCK_SIZE), just in case someone else update it to a weird value later. ------------- Marked as reviewed by xuelei (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/411 From mcimadamore at openjdk.java.net Wed Oct 7 17:30:38 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Wed, 7 Oct 2020 17:30:38 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) Message-ID: This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html Specdiff: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html CSR: https://bugs.openjdk.java.net/browse/JDK-8254163 ### API Changes * `MemorySegment` * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) * added a no-arg factory for a native restricted segment representing entire native heap * rename `withOwnerThread` to `handoff` * add new `share` method, to create shared segments * add new `registerCleaner` method, to register a segment against a cleaner * add more helpers to create arrays from a segment e.g. `toIntArray` * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) * `MemoryAddress` * drop `segment` accessor * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment * `MemoryAccess` * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). * `MemoryHandles` * drop `withOffset` combinator * drop `withStride` combinator * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. * `Addressable` * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. * `MemoryLayouts` * A new layout, for machine addresses, has been added to the mix. ### Implementation changes There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. #### Shared segments The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### Memory access var handles overhaul The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 ------------- Commit messages: - Add modified files - RFR 8254162: Implementation of Foreign-Memory Access API (Third Incubator) Changes: https://git.openjdk.java.net/jdk/pull/548/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8254162 Stats: 7467 lines in 75 files changed: 5024 ins; 1373 del; 1070 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From valeriep at openjdk.java.net Wed Oct 7 19:19:11 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 7 Oct 2020 19:19:11 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM In-Reply-To: References: Message-ID: <9IvsxvHGzvQoM756tUBFHLcMyZjvdbevzh7c-I3i6zU=.03c89dcb-a547-4a07-b0db-358fa9b3fe27@github.com> On Tue, 29 Sep 2020 20:22:55 GMT, Anthony Scarpino wrote: > 8253821: Improve ByteBuffer performance with GCM I will take a look as well. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From erikj at openjdk.java.net Thu Oct 8 06:50:04 2020 From: erikj at openjdk.java.net (Erik Joelsson) Date: Thu, 8 Oct 2020 06:50:04 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) In-Reply-To: References: Message-ID: <_GsbJIUKoZcZhBL3g5LIazayukM61sEJOqQcKpp7Hzw=.12a07196-262a-411e-bb04-70e1321564dd@github.com> On Wed, 7 Oct 2020 17:13:22 GMT, Maurizio Cimadamore wrote: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Build changes look pretty good, just a few minor nits. make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 33: > 31: > 32: SCOPED_MEMORY_ACCESS_TEMPLATE := > $(TOPDIR)/src/java.base/share/classes/jdk/internal/misc/X-ScopedMemoryAccess.java.template 33: > SCOPED_MEMORY_ACCESS_BIN_TEMPLATE := > $(TOPDIR)/src/java.base/share/classes/jdk/internal/misc/X-ScopedMemoryAccess-bin.java.template Should these variables be defined based on SCOPED_MEMORY_ACCESS_SRC_DIR to avoid repeating that path? make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 151: > 149: $(CP) $(SCOPED_MEMORY_ACCESS_TEMPLATE) $(DEST) > 150: $(foreach t, $(SCOPE_MEMORY_ACCESS_TYPES), \ > 151: $(TOOL_SPP) -nel -K$(BIN_$t_type) -Dtype=$(BIN_$t_type) -DType=$(BIN_$t_Type) $(BIN_$t_ARGS) \ Please indent with and 2 spaces for logical indent of the foreach body. make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 155: > 153: $(PRINTF) "}\n" >> $(DEST) > 154: > 155: TARGETS += $(DEST) Missing newline. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From hchao at openjdk.java.net Thu Oct 8 06:50:02 2020 From: hchao at openjdk.java.net (Hai-May Chao) Date: Thu, 8 Oct 2020 06:50:02 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> On Thu, 1 Oct 2020 20:02:34 GMT, Weijun Wang wrote: > Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at > https://bugs.openjdk.java.net/browse/JDK-8228481. Looks good. Only minor comments. src/java.base/share/classes/sun/security/pkcs12/PKCS12KeyStore.java line 103: > 101: = "PBEWithHmacSHA256AndAES_256"; > 102: private static final String DEFAULT_MAC_ALGORITHM = "HmacPBESHA256"; > 103: private static final int DEFAULT_PBE_ITERATION_COUNT = 50000; As we have keystore.pkcs12.certPbeIterationCount and keystore.pkcs12.keyPbeIterationCount, I would like to suggest that we can define DEFAULT_CERT_PBE_ITERATION_COUNT and DEFAULT_KEY_PBE_ITERATION_COUNT, specifying each of the values for finer granularity. Same for LEGACY_PBE_ITERATION_COUNT. test/jdk/sun/security/mscapi/VeryLongAlias.java line 48: > 46: > 47: static String alias = String.format("%0512d", new Random().nextInt(100000)); > 48: Add bug number to @bug. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From hchao at openjdk.java.net Thu Oct 8 06:50:18 2020 From: hchao at openjdk.java.net (Hai-May Chao) Date: Thu, 8 Oct 2020 06:50:18 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> Message-ID: <-yG4ohed5byKufPfUr8_fg3W8GKVL_v0ZFNgx8o718U=.fea342a6-b204-4b84-bc13-e7837a61aed5@github.com> On Wed, 7 Oct 2020 22:08:19 GMT, Hai-May Chao wrote: >> Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at >> https://bugs.openjdk.java.net/browse/JDK-8228481. > > Looks good. Only minor comments. CSR looks good. In "Sepcification" section: a typo in 'Thr iteration counts used by'. At the end, it describes the new system property will override the security properties and use the older and weaker algorithms, so suggest we could also add text about setting the iteration counts to the default legacy values. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Thu Oct 8 06:52:03 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 8 Oct 2020 06:52:03 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> Message-ID: <3SxxYgJscCh_tYc5l2D7ko3FWS2RxxPvbMend12Xc58=.0ab04535-b6ae-4967-8aa5-c3d4c0484695@github.com> On Wed, 7 Oct 2020 22:06:28 GMT, Hai-May Chao wrote: >> Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at >> https://bugs.openjdk.java.net/browse/JDK-8228481. > > test/jdk/sun/security/mscapi/VeryLongAlias.java line 48: > >> 46: >> 47: static String alias = String.format("%0512d", new Random().nextInt(100000)); >> 48: > > Add bug number to @bug. OK. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Thu Oct 8 06:50:31 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 8 Oct 2020 06:50:31 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <-yG4ohed5byKufPfUr8_fg3W8GKVL_v0ZFNgx8o718U=.fea342a6-b204-4b84-bc13-e7837a61aed5@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> <-yG4ohed5byKufPfUr8_fg3W8GKVL_v0ZFNgx8o718U=.fea342a6-b204-4b84-bc13-e7837a61aed5@github.com> Message-ID: On Wed, 7 Oct 2020 22:20:07 GMT, Hai-May Chao wrote: >> Looks good. Only minor comments. > > CSR looks good. In "Sepcification" section: a typo in 'Thr iteration counts used by'. At the end, it describes the new > system property will override the security properties and use the older and weaker algorithms, so suggest we could also > add text about setting the iteration counts to the default legacy values. CSR updated. More description, and iteration counts lowered to 10000. Will update code soon. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From jpai at openjdk.java.net Thu Oct 8 06:54:57 2020 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 8 Oct 2020 06:54:57 GMT Subject: RFR: 8242882: opening jar file with large manifest might throw NegativeArraySizeException [v3] In-Reply-To: <-1TnPxiL-jvYX34VVaIpduNdE687CpEGm_65B4tb-P8=.e5983c5f-405a-4629-b232-a6b284d9d95a@github.com> References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> <9AgaGLA8q63mDTKDeavbyfZVntagI0bd0Kb7rkQQYyg=.67d4410c-b7db-4048-87d0-4ec1f9d93cd2@github.com> <-1TnPxiL-jvYX34VVaIpduNdE687CpEGm_65B4tb-P8=.e5983c5f-405a-4629-b232-a6b284d9d95a@github.com> Message-ID: On Wed, 7 Oct 2020 21:40:57 GMT, Brent Christian wrote: >> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: >> >> Second round of review comments addressed > > Marked as reviewed by bchristi (Reviewer). Hello Lance, does the latest state of this PR look fine to you? If so, shall I trigger a integrate? ------------- PR: https://git.openjdk.java.net/jdk/pull/323 From bchristi at openjdk.java.net Thu Oct 8 06:54:40 2020 From: bchristi at openjdk.java.net (Brent Christian) Date: Thu, 8 Oct 2020 06:54:40 GMT Subject: RFR: 8242882: opening jar file with large manifest might throw NegativeArraySizeException [v3] In-Reply-To: <9AgaGLA8q63mDTKDeavbyfZVntagI0bd0Kb7rkQQYyg=.67d4410c-b7db-4048-87d0-4ec1f9d93cd2@github.com> References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> <9AgaGLA8q63mDTKDeavbyfZVntagI0bd0Kb7rkQQYyg=.67d4410c-b7db-4048-87d0-4ec1f9d93cd2@github.com> Message-ID: <-1TnPxiL-jvYX34VVaIpduNdE687CpEGm_65B4tb-P8=.e5983c5f-405a-4629-b232-a6b284d9d95a@github.com> On Thu, 1 Oct 2020 14:42:21 GMT, Jaikiran Pai wrote: >> Can I please get a review and a sponsor for a fix for https://bugs.openjdk.java.net/browse/JDK-8242882? >> >> As noted in that JBS issue, if the size of the Manifest entry in the jar happens to be very large (such that it exceeds >> the `Integer.MAX_VALUE`), then the current code in `JarFile#getBytes` can lead to a `NegativeArraySizeException`. This >> is due to the: if (len != -1 && len <= 65535) block which evaluates to `true` when the size of the manifest entry is >> larger than `Integer.MAX_VALUE`. As a result, this then ends up calling the code which can lead to the >> `NegativeArraySizeException`. The commit in this PR fixes that issue by changing those `if/else` blocks to prevent >> this issue and instead use a code path that leads to the `InputStream#readAllBytes()` which internally has the >> necessary checks to throw the expected `OutOfMemoryError`. This commit also includes a jtreg test case which >> reproduces the issue and verifies the fix. > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > Second round of review comments addressed Marked as reviewed by bchristi (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/323 From bchristi at openjdk.java.net Thu Oct 8 06:55:24 2020 From: bchristi at openjdk.java.net (Brent Christian) Date: Thu, 8 Oct 2020 06:55:24 GMT Subject: RFR: 8242882: opening jar file with large manifest might throw NegativeArraySizeException [v2] In-Reply-To: References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> Message-ID: On Thu, 1 Oct 2020 14:39:50 GMT, Jaikiran Pai wrote: >> test/jdk/java/util/jar/JarFile/LargeManifestOOMTest.java line 78: >> >>> 76: bw.write("OOM-Test: "); >>> 77: for (long i = 0; i < 2147483648L; i++) { >>> 78: bw.write("a"); >> >> As you probably noticed, this test takes a little while to run. One way to speed it up a little would be to write more >> characters at a time. While we're at it, we may as well make the Manifest well-formed by breaking it into 72-byte >> lines. See "Line length" under: >> https://docs.oracle.com/en/java/javase/15/docs/specs/jar/jar.html#notes-on-manifest-and-signature-files Just write >> enough lines to exceed Integer.MAX_VALUE bytes. > > I decided to slightly change the way this large manifest file was being created. I borrowed the idea from > `Zip64SizeTest`[1] to create the file and set its length to a large value. I hope that is OK. If not, let me know, I > will change this part. [1] > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/Zip64SizeTest.java#L121 I did some automated test runs, and the duration of this test is sufficiently improved, IMO. While not representative of a real MANIFEST.MF file, I think it works well enough for this specific test. ------------- PR: https://git.openjdk.java.net/jdk/pull/323 From jpai at openjdk.java.net Thu Oct 8 06:55:32 2020 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 8 Oct 2020 06:55:32 GMT Subject: RFR: 8242882: opening jar file with large manifest might throw NegativeArraySizeException [v2] In-Reply-To: References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> Message-ID: On Wed, 7 Oct 2020 21:40:43 GMT, Brent Christian wrote: >> I decided to slightly change the way this large manifest file was being created. I borrowed the idea from >> `Zip64SizeTest`[1] to create the file and set its length to a large value. I hope that is OK. If not, let me know, I >> will change this part. [1] >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/Zip64SizeTest.java#L121 > > I did some automated test runs, and the duration of this test is sufficiently improved, IMO. > > While not representative of a real MANIFEST.MF file, I think it works well enough for this specific test. Thank you for the review Brent. ------------- PR: https://git.openjdk.java.net/jdk/pull/323 From ascarpino at openjdk.java.net Thu Oct 8 06:51:08 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 06:51:08 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: > 8253821: Improve ByteBuffer performance with GCM Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: Xuelei comments ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/411/files - new: https://git.openjdk.java.net/jdk/pull/411/files/3a3a0296..b29c1ed5 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=411&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=411&range=00-01 Stats: 17 lines in 6 files changed: 6 ins; 7 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/411.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/411/head:pull/411 PR: https://git.openjdk.java.net/jdk/pull/411 From valeriep at openjdk.java.net Thu Oct 8 06:53:39 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Thu, 8 Oct 2020 06:53:39 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 03:21:46 GMT, Anthony Scarpino wrote: >> 8253821: Improve ByteBuffer performance with GCM > > Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: > > Xuelei comments src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 658: > 656: BadPaddingException { > 657: return bufferCrypt(input, output, false); > 658: } Is the override of this method for using a different bufferCrypt impl? There is also engineUpdate(ByteBuffer, ByteBuffer) in CipherSpi, is there a reason for not overriding that here? src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 744: > 742: } else { > 743: return core.doFinal(input, output); > 744: } It seems this block is the only difference between this method and CipherSpi.bufferCrypt(). Have you considered moving this special handling to the overridden engineDoFinal(...) method and not duplicating the whole CipherSpi.bufferCrypt() method here? BTW, instead of using the generic update/doFinal name and then commenting them for GCM usage only, perhaps it's more enticing to name them as gcmUpdate/gcmDoFinal? src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 820: > 818: return cipher.decrypt(src, dst); > 819: } > 820: return cipher.encrypt(src, dst); How about return (decrypting? cipher.decrypt(src, dst) : cipher.encrypt(src, dst)); src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 816: > 814: } > 815: > 816: int update(ByteBuffer src, ByteBuffer dst) throws ShortBufferException { Is this also one of the "GCM only" methods? If so, same comment as doFinal(ByteBuffer, ByteBuffer)? Maybe the name should be more specific to avoid misuse. src/java.base/share/classes/com/sun/crypto/provider/FeedbackCipher.java line 261: > 259: > 260: int encryptFinal(ByteBuffer src, ByteBuffer dst) > 261: throws IllegalBlockSizeException, AEADBadTagException, Tag is generated during encryption, this can't possibly throw AEADBadTagException, copy-n-paste error maybe? src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 1253: > 1251: if (decrypting) { > 1252: if (buffered > 0) { > 1253: cipher.decrypt(buffer, 0, buffered, new byte[0], 0); This looks a bit strange? The output buffer is 0-length which would lead to ShortBufferException when the buffered bytes is enough to produce some output. src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 1258: > 1256: } else { > 1257: if (buffered > 0) { > 1258: cipher.encrypt(buffer, 0, buffered, new byte[0], 0); Same comment as above? src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 939: > 937: // if the size of specified output buffer is less than > 938: // the length of the cipher text, then the current > 939: // content of cipher has to be preserved in order for This change is somewhat dangerous. When using the supplied output buffer directly, you may corrupt its content w/ padding bytes. This optimization should only be applied when no padding is involved. In addition, input and output can point to the same buffer with all sorts of index combinations, i.e. inOfs == outOfs, inOfs < outOfs, inOfs > outOfs. With "outWithPadding" approach, no need to check. However, for "internalOutput", data corruption may happen. This kind of problem can be very hard to diagnose. Have to be very very careful here as this code may impact all ciphers... src/java.base/share/classes/com/sun/crypto/provider/GCTR.java line 240: > 238: } > 239: } > 240: } See the above red icon? It warns about missing newline at end of this file. src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 528: > 526: } > 527: > 528: ArrayUtil.blockSizeCheck(src.remaining(), blockSize); Hmm, I am not sure if this check still applies in ByteBuffer case. You are passing the ByteBuffer objs directly from AESCipher->CipherCore->GaloisCounterMode. This is different from the byte[] case where CipherCore would chop up the data into blocks and pass the blocks to the underlying FeedbackCipher impl. Perhaps no existing regression tests covers ByteBuffer inputs w/ non-blocksize data? Otherwise, this should be caught? BTW, why not just use 'len' again? Seems unnecessary to keep calling src.remaining() in various places in this method. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From valeriep at openjdk.java.net Thu Oct 8 06:54:16 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Thu, 8 Oct 2020 06:54:16 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: <_WNpCbfyAeN1LTEDZhSv_YRGnPAVUsP6jES-PZSzCp4=.c97c07af-f8c1-4562-9e8a-c91909b2a5ed@github.com> On Wed, 7 Oct 2020 16:17:38 GMT, Xue-Lei Andrew Fan wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 664: > >> 662: * engineUpdate() and engineDoFinal(). >> 663: */ >> 664: private int bufferCrypt(ByteBuffer input, ByteBuffer output, > > It looks like this method is copied from the CipherSpi. For maintenance, it would be nice to add an extra comment to > state the copy and update. For example, "this method and implementation is copied from javax.crypto.CipherSpi, with an > improvement for GCM mode." Agree w/ Xuelei, it'd be nice to mention CipherSpi.bufferCrypt() method. In case a bug is found, it'd be checked and fixed in both places. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Thu Oct 8 06:54:36 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 06:54:36 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <_WNpCbfyAeN1LTEDZhSv_YRGnPAVUsP6jES-PZSzCp4=.c97c07af-f8c1-4562-9e8a-c91909b2a5ed@github.com> References: <_WNpCbfyAeN1LTEDZhSv_YRGnPAVUsP6jES-PZSzCp4=.c97c07af-f8c1-4562-9e8a-c91909b2a5ed@github.com> Message-ID: On Wed, 7 Oct 2020 19:42:11 GMT, Valerie Peng wrote: >> src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 664: >> >>> 662: * engineUpdate() and engineDoFinal(). >>> 663: */ >>> 664: private int bufferCrypt(ByteBuffer input, ByteBuffer output, >> >> It looks like this method is copied from the CipherSpi. For maintenance, it would be nice to add an extra comment to >> state the copy and update. For example, "this method and implementation is copied from javax.crypto.CipherSpi, with an >> improvement for GCM mode." > > Agree w/ Xuelei, it'd be nice to mention CipherSpi.bufferCrypt() method. In case a bug is found, it'd be checked and > fixed in both places. Yeah.. good point. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Thu Oct 8 06:55:34 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 06:55:34 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: <5pNHm_DfFSouqsnzLP_ADJOtn7tieyUsOjKY1OjJIVs=.ae872784-b379-4283-ac9b-67c3475d02bb@github.com> On Wed, 7 Oct 2020 16:29:32 GMT, Xue-Lei Andrew Fan wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 947: > >> 945: // create temporary output buffer if the estimated size is larger >> 946: // than the user-provided buffer. >> 947: if (output.length - outputOffset < estOutSize) { > > "outputCapacity" could be used to replace "output.length - outputOffset", and join the clause with the if-clause above. interesting.. That makes this and the if condition below it able to join again. > src/java.base/share/classes/com/sun/crypto/provider/CounterMode.java line 28: > >> 26: package com.sun.crypto.provider; >> 27: >> 28: import java.nio.ByteBuffer; > > There is no more update except this line. The new import ByteBuffer may not be used. Thanks. That is another new thing to get used to with git and multiple commits. When cleaning up code it's easy to not see changes like this. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Thu Oct 8 06:56:44 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 06:56:44 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Wed, 7 Oct 2020 22:38:21 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 939: > >> 937: // if the size of specified output buffer is less than >> 938: // the length of the cipher text, then the current >> 939: // content of cipher has to be preserved in order for > > This change is somewhat dangerous. When using the supplied output buffer directly, you may corrupt its content w/ > padding bytes. This optimization should only be applied when no padding is involved. In addition, input and output can > point to the same buffer with all sorts of index combinations, i.e. inOfs == outOfs, inOfs < outOfs, inOfs > outOfs. > With "outWithPadding" approach, no need to check. However, for "internalOutput", data corruption may happen. This kind > of problem can be very hard to diagnose. Have to be very very careful here as this code may impact all ciphers... - I do not understand where the corruption comes from. The user provides a buffer that output is suppose to be placed into, what could it be corrupting? The existing tests (SameBuffer, in particular) works fine with this and the ByteBuffer calls. I spent a lot of time trying to get those same buffer tests to pass. - It was my expectation that checkOutputCapacity() is making sure all the inOfs ==<> outOfs are going to work. Does that not catch all cases? - outWithPadding" is excessive because it doubles the allocation for every operation followed by a copy to the original buffer, even if the original buffer was adequate. I'm ok with doing the extra alloc & copy in certain situations, but not everytime. Can you be more specific what things may fail that we don't already check for in the regression tests? > src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 658: > >> 656: BadPaddingException { >> 657: return bufferCrypt(input, output, false); >> 658: } > > Is the override of this method for using a different bufferCrypt impl? There is also engineUpdate(ByteBuffer, > ByteBuffer) in CipherSpi, is there a reason for not overriding that here? Yes. thanks.. The IDE covered that up by going to the one in CipherSpi and I thought it was calling the AES one. TLS doesn't use update() so the perf numbers won't change. I'll have to run the tests again. > src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 744: > >> 742: } else { >> 743: return core.doFinal(input, output); >> 744: } > > It seems this block is the only difference between this method and CipherSpi.bufferCrypt(). Have you considered moving > this special handling to the overridden engineDoFinal(...) method and not duplicating the whole CipherSpi.bufferCrypt() > method here? BTW, instead of using the generic update/doFinal name and then commenting them for GCM usage only, perhaps > it's more enticing to name them as gcmUpdate/gcmDoFinal? I didn't see a way to override this because CipherSpi is a public class, any methods I added would become a new API. Also bufferCrypt is private, so I had to copy it. CipherSpi does not know which mode is being used, but AESCipher does. Maybe I'm missing something, I'd rather it not be a copy, but I couldn't see a better way. If you have a specific idea, please give me details. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From shade at openjdk.java.net Thu Oct 8 07:44:43 2020 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Thu, 8 Oct 2020 07:44:43 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) In-Reply-To: References: Message-ID: <2m-ZUtCAOMDADZVduYjzmcNHYziHMIC3axl1CNolulo=.4e19ccf9-4320-434d-b8c3-b186f74f1b5e@github.com> On Wed, 7 Oct 2020 17:13:22 GMT, Maurizio Cimadamore wrote: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Drive-by review. src/hotspot/share/prims/scopedMemoryAccess.cpp line 1: > 1: Misses copyright header. src/hotspot/share/prims/scopedMemoryAccess.cpp line 81: > 79: CompiledMethod* cm = last_frame.cb()->as_compiled_method(); > 80: > 81: //FIXME: this doesn't work if reachability fences are violated by C2 Maybe turn this into /* */ block, so that it is obvious the whole thing relates to the `FIXME`? src/hotspot/share/prims/scopedMemoryAccess.cpp line 96: > 94: int depth = 0; > 95: vframeStream stream(jt); > 96: for (; !stream.at_end(); stream.next()) { Can move `stream` initialization right into the `for` statement? src/hotspot/share/prims/scopedMemoryAccess.cpp line 138: > 136: /// JVM_RegisterUnsafeMethods > 137: > 138: #define LANG "Ljdk/internal/misc/" This is weirdly defined as `LANG`. I suppose this would change to `Ljava/lang` soon. But maybe `PACKAGE` is better. src/hotspot/share/prims/scopedMemoryAccess.cpp line 130: > 128: * Top frames containg obj will be deoptimized. > 129: */ > 130: JVM_ENTRY(jboolean, ScopedMemoryAccess_closeScope(JNIEnv *env, jobject receiver, jobject deopt, jobject > exception)) { `JVM_ENTRY` does not require a brace, it is braced already. See existing uses of `JVM_ENTRY`. src/hotspot/share/prims/scopedMemoryAccess.cpp line 134: > 132: Handshake::execute(&cl); > 133: return !cl._found; > 134: } JVM_END Ditto for `JVM_END`. src/hotspot/share/prims/scopedMemoryAccess.cpp line 166: > 164: int ok = env->RegisterNatives(scopedMemoryAccessClass, jdk_internal_misc_ScopedMemoryAccess_methods, > sizeof(jdk_internal_misc_ScopedMemoryAccess_methods)/sizeof(JNINativeMethod)); 165: guarantee(ok == 0, "register > jdk.internal.misc.ScopedMemoryAccess natives"); 166: } JVM_END `JVM_ENTRY`/`JVM_END` braces again. src/java.base/share/classes/java/lang/invoke/MemoryAccessVarHandleBase.java line 45: > 43: final boolean skipAlignmentMaskCheck; > 44: > 45: MemoryAccessVarHandleBase(VarForm form, boolean skipOffetCheck, boolean be, long length, long alignmentMask) { Typo: `skipOff*s*etCheck`. Should it be `skipAlignmentMaskCheck` to begin with? test/jdk/java/foreign/TestMismatch.java line 26: > 24: /* > 25: * @test > 26: * @run testng/othervm -XX:MaxDirectMemorySize=5000000000 TestMismatch Whoa, allocating 5 GB? That might fail on 32-bit platforms... Anyhow, this flag accepts suffixes, so `-XX:MaxDirectMemorySize=5g`. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Thu Oct 8 10:29:24 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 8 Oct 2020 10:29:24 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v2] In-Reply-To: References: Message-ID: <-K8a4gV16AZ7Se7-G2DWZrSEMr5FjLPzlUlo4nXnTE0=.c33f2a73-6ee1-4a9f-b992-d51fc1f2f481@github.com> > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Address review comments ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/e4eb2c74..fa051abf Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=00-01 Stats: 67 lines in 5 files changed: 33 ins; 3 del; 31 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Thu Oct 8 10:29:25 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 8 Oct 2020 10:29:25 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v2] In-Reply-To: <2m-ZUtCAOMDADZVduYjzmcNHYziHMIC3axl1CNolulo=.4e19ccf9-4320-434d-b8c3-b186f74f1b5e@github.com> References: <2m-ZUtCAOMDADZVduYjzmcNHYziHMIC3axl1CNolulo=.4e19ccf9-4320-434d-b8c3-b186f74f1b5e@github.com> Message-ID: On Thu, 8 Oct 2020 06:53:41 GMT, Aleksey Shipilev wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Address review comments > > test/jdk/java/foreign/TestMismatch.java line 26: > >> 24: /* >> 25: * @test >> 26: * @run testng/othervm -XX:MaxDirectMemorySize=5000000000 TestMismatch > > Whoa, allocating 5 GB? That might fail on 32-bit platforms... Anyhow, this flag accepts suffixes, so > `-XX:MaxDirectMemorySize=5g`. I've done two things here: * the limit isn't really doing much in this test, so I've removed * I moved the limit in TestSegments; the limit is set to much lower threshold (2M) which should work regardless of 32/64 * For TestMismatch, which needs to allocate a segment bigger than 2^32 in one of the tests, I've added a guard in the offending test which verifies that we're indeed on a 64-bit platform ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From shade at openjdk.java.net Thu Oct 8 10:32:49 2020 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Thu, 8 Oct 2020 10:32:49 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v2] In-Reply-To: <-K8a4gV16AZ7Se7-G2DWZrSEMr5FjLPzlUlo4nXnTE0=.c33f2a73-6ee1-4a9f-b992-d51fc1f2f481@github.com> References: <-K8a4gV16AZ7Se7-G2DWZrSEMr5FjLPzlUlo4nXnTE0=.c33f2a73-6ee1-4a9f-b992-d51fc1f2f481@github.com> Message-ID: On Thu, 8 Oct 2020 10:29:24 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation >> (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from >> multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee >> that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class >> has been added, which defines several useful dereference routines; these are really just thin wrappers around memory >> access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not >> the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link >> to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit >> of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which >> wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as >> dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability >> in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; >> secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can >> use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done >> by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided >> below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be >> happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, >> Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd >> like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio >> Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative >> to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a >> carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access >> base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte >> offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. >> `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which >> it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both >> `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients >> can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory >> access var handle support. >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to >> achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment >> is shared, it would be possible for a thread to close it while another is accessing it. After considering several >> options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he >> reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world >> (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a >> close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and >> the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). >> It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of >> these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, >> we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to >> whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of >> stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, >> once we detect that a thread is accessing the very segment we're about to close, what should happen? We first >> experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it >> fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the >> machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to >> minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread >> is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and >> try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should >> be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single >> place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in >> addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` >> annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) >> class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, >> like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is >> tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during >> access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory >> access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead >> of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a >> `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed >> successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two >> implementations, one for confined segments and one for shared segments; the main difference between the two is what >> happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared >> segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or >> `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` >> state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### >> Memory access var handles overhaul The key realization here was that if all memory access var handles took a >> coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle >> form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var >> handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that >> e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the >> implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level >> access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see >> here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, >> since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` >> functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the >> microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared >> segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - >> https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Address review comments Thanks, these changes make sense to me. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From lancea at openjdk.java.net Thu Oct 8 10:35:45 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Thu, 8 Oct 2020 10:35:45 GMT Subject: RFR: 8242882: opening jar file with large manifest might throw NegativeArraySizeException [v3] In-Reply-To: <9AgaGLA8q63mDTKDeavbyfZVntagI0bd0Kb7rkQQYyg=.67d4410c-b7db-4048-87d0-4ec1f9d93cd2@github.com> References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> <9AgaGLA8q63mDTKDeavbyfZVntagI0bd0Kb7rkQQYyg=.67d4410c-b7db-4048-87d0-4ec1f9d93cd2@github.com> Message-ID: On Thu, 1 Oct 2020 14:42:21 GMT, Jaikiran Pai wrote: >> Can I please get a review and a sponsor for a fix for https://bugs.openjdk.java.net/browse/JDK-8242882? >> >> As noted in that JBS issue, if the size of the Manifest entry in the jar happens to be very large (such that it exceeds >> the `Integer.MAX_VALUE`), then the current code in `JarFile#getBytes` can lead to a `NegativeArraySizeException`. This >> is due to the: if (len != -1 && len <= 65535) block which evaluates to `true` when the size of the manifest entry is >> larger than `Integer.MAX_VALUE`. As a result, this then ends up calling the code which can lead to the >> `NegativeArraySizeException`. The commit in this PR fixes that issue by changing those `if/else` blocks to prevent >> this issue and instead use a code path that leads to the `InputStream#readAllBytes()` which internally has the >> necessary checks to throw the expected `OutOfMemoryError`. This commit also includes a jtreg test case which >> reproduces the issue and verifies the fix. > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > Second round of review comments addressed Hi Jaikiran, Yes I think you are OK to move forward with the integration, ------------- Marked as reviewed by lancea (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/323 From jpai at openjdk.java.net Thu Oct 8 10:49:51 2020 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 8 Oct 2020 10:49:51 GMT Subject: Integrated: 8242882: opening jar file with large manifest might throw NegativeArraySizeException In-Reply-To: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> References: <17lthSuYGInBHE2r3hBs0yXvMIZWWQkdLhYYBfRUMfM=.b59db398-6dc4-4b34-b141-2c58f189bce8@github.com> Message-ID: On Wed, 23 Sep 2020 15:06:55 GMT, Jaikiran Pai wrote: > Can I please get a review and a sponsor for a fix for https://bugs.openjdk.java.net/browse/JDK-8242882? > > As noted in that JBS issue, if the size of the Manifest entry in the jar happens to be very large (such that it exceeds > the `Integer.MAX_VALUE`), then the current code in `JarFile#getBytes` can lead to a `NegativeArraySizeException`. This > is due to the: if (len != -1 && len <= 65535) block which evaluates to `true` when the size of the manifest entry is > larger than `Integer.MAX_VALUE`. As a result, this then ends up calling the code which can lead to the > `NegativeArraySizeException`. The commit in this PR fixes that issue by changing those `if/else` blocks to prevent > this issue and instead use a code path that leads to the `InputStream#readAllBytes()` which internally has the > necessary checks to throw the expected `OutOfMemoryError`. This commit also includes a jtreg test case which > reproduces the issue and verifies the fix. This pull request has now been integrated. Changeset: 782d45bd Author: Jaikiran Pai Committer: Lance Andersen URL: https://git.openjdk.java.net/jdk/commit/782d45bd Stats: 85 lines in 2 files changed: 84 ins; 0 del; 1 mod 8242882: opening jar file with large manifest might throw NegativeArraySizeException Reviewed-by: bchristi, lancea ------------- PR: https://git.openjdk.java.net/jdk/pull/323 From erikj at openjdk.java.net Thu Oct 8 12:56:43 2020 From: erikj at openjdk.java.net (Erik Joelsson) Date: Thu, 8 Oct 2020 12:56:43 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v2] In-Reply-To: <-K8a4gV16AZ7Se7-G2DWZrSEMr5FjLPzlUlo4nXnTE0=.c33f2a73-6ee1-4a9f-b992-d51fc1f2f481@github.com> References: <-K8a4gV16AZ7Se7-G2DWZrSEMr5FjLPzlUlo4nXnTE0=.c33f2a73-6ee1-4a9f-b992-d51fc1f2f481@github.com> Message-ID: On Thu, 8 Oct 2020 10:29:24 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation >> (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from >> multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee >> that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class >> has been added, which defines several useful dereference routines; these are really just thin wrappers around memory >> access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not >> the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link >> to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit >> of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which >> wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as >> dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability >> in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; >> secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can >> use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done >> by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided >> below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be >> happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, >> Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd >> like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio >> Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative >> to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a >> carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access >> base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte >> offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. >> `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which >> it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both >> `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients >> can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory >> access var handle support. >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to >> achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment >> is shared, it would be possible for a thread to close it while another is accessing it. After considering several >> options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he >> reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world >> (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a >> close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and >> the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). >> It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of >> these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, >> we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to >> whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of >> stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, >> once we detect that a thread is accessing the very segment we're about to close, what should happen? We first >> experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it >> fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the >> machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to >> minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread >> is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and >> try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should >> be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single >> place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in >> addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` >> annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) >> class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, >> like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is >> tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during >> access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory >> access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead >> of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a >> `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed >> successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two >> implementations, one for confined segments and one for shared segments; the main difference between the two is what >> happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared >> segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or >> `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` >> state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### >> Memory access var handles overhaul The key realization here was that if all memory access var handles took a >> coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle >> form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var >> handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that >> e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the >> implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level >> access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see >> here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, >> since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` >> functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the >> microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared >> segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - >> https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Address review comments make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 145: > 143: SCOPE_MEMORY_ACCESS_TYPES := Byte Short Char Int Long Float Double > 144: $(foreach t, $(SCOPE_MEMORY_ACCESS_TYPES), \ > 145: $(eval $(call GenerateScopedOp,BIN_$t,$t))) This indent was fine at 2 spaces. I meant the one below inside the recipe. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Thu Oct 8 13:49:47 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 8 Oct 2020 13:49:47 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v2] In-Reply-To: References: <-K8a4gV16AZ7Se7-G2DWZrSEMr5FjLPzlUlo4nXnTE0=.c33f2a73-6ee1-4a9f-b992-d51fc1f2f481@github.com> Message-ID: On Thu, 8 Oct 2020 12:54:12 GMT, Erik Joelsson wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Address review comments > > make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 145: > >> 143: SCOPE_MEMORY_ACCESS_TYPES := Byte Short Char Int Long Float Double >> 144: $(foreach t, $(SCOPE_MEMORY_ACCESS_TYPES), \ >> 145: $(eval $(call GenerateScopedOp,BIN_$t,$t))) > > This indent was fine at 2 spaces. I meant the one below inside the recipe. Gotcha - I fixed the wrong foreach... ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Thu Oct 8 13:59:20 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 8 Oct 2020 13:59:20 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v3] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Fix indent in GensrcScopedMemoryAccess.gmk ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/fa051abf..b941c4a2 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=01-02 Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From weijun at openjdk.java.net Thu Oct 8 14:23:57 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 8 Oct 2020 14:23:57 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v2] In-Reply-To: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: > Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at > https://bugs.openjdk.java.net/browse/JDK-8228481. Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: change ic to 10000 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/473/files - new: https://git.openjdk.java.net/jdk/pull/473/files/b99611b3..6b5c5b5e Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=473&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=473&range=00-01 Stats: 52 lines in 5 files changed: 1 ins; 1 del; 50 mod Patch: https://git.openjdk.java.net/jdk/pull/473.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/473/head:pull/473 PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Thu Oct 8 14:23:57 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 8 Oct 2020 14:23:57 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v2] In-Reply-To: References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> <-yG4ohed5byKufPfUr8_fg3W8GKVL_v0ZFNgx8o718U=.fea342a6-b204-4b84-bc13-e7837a61aed5@github.com> Message-ID: On Wed, 7 Oct 2020 22:49:09 GMT, Weijun Wang wrote: >> CSR looks good. In "Sepcification" section: a typo in 'Thr iteration counts used by'. At the end, it describes the new >> system property will override the security properties and use the older and weaker algorithms, so suggest we could also >> add text about setting the iteration counts to the default legacy values. > > CSR updated. More description, and iteration counts lowered to 10000. Will update code soon. New commit updating ic to 10000. I also created separate constants for DEFAULT_CERT_PBE_ITERATION_COUNT and DEFAULT_KEY_PBE_ITERATION_COUNT. I haven't made the change for LEGACY_PBE_ITERATION_COUNT since they will never change. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From dfuchs at openjdk.java.net Thu Oct 8 15:24:44 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Thu, 8 Oct 2020 15:24:44 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers Message-ID: Hi, This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of synchronized monitors. Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are outlined below: 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing ( a snapshot of the data is taken before printing instead) 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: `MeteredStream::close` was missing synchronization: it is now protected by the lock 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of casting to PrintStream. 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. There is no need for a synchronized here as the method is only called from a method that already holds the lock. 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. This method is only called from methods already protected by the lock. Also the method has been made private. 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java `queuedForCleanup` is made volatile as it is read and written directly from outside the class and without protection by the `KeepAliveCleanerEntry`. Lock protection is also added to `close()`, which was missing it. Some methods that have no lock protection and did not need it because only called from within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` to help track the fact that AuthenticationInfo::setHeaders is only called while holding the lock. Synchronization was also removed from some method that didn't need it because only called from within code blocks protected by the lock: `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: synchronization removed from `setHeaders` and replace by an assert. See point 7. above. 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: synchronization removed from `setHeaders` and replace by an assert. See point 7. above. 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: synchronization removed from `setHeaders` and replace by an assert. See point 7. above. ------------- Commit messages: - 8229867: Re-examine synchronization usages in http and https protocol handlers Changes: https://git.openjdk.java.net/jdk/pull/558/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=558&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8229867 Stats: 1116 lines in 18 files changed: 551 ins; 149 del; 416 mod Patch: https://git.openjdk.java.net/jdk/pull/558.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/558/head:pull/558 PR: https://git.openjdk.java.net/jdk/pull/558 From ascarpino at openjdk.java.net Thu Oct 8 15:29:56 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 15:29:56 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: <7ALsHg_fkNnQ_BFpdYZ7zK8Sm2rHli8fvZK99cpTfbk=.4dfe0582-d202-43fe-8869-c882f765bc5d@github.com> On Wed, 7 Oct 2020 20:34:21 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 816: > >> 814: } >> 815: >> 816: int update(ByteBuffer src, ByteBuffer dst) throws ShortBufferException { > > Is this also one of the "GCM only" methods? If so, same comment as doFinal(ByteBuffer, ByteBuffer)? > Maybe the name should be more specific to avoid misuse. Ok.. I see what you mean by renaming the method. Yeah, I suppose it's better since it's not truely generic > src/java.base/share/classes/com/sun/crypto/provider/FeedbackCipher.java line 261: > >> 259: >> 260: int encryptFinal(ByteBuffer src, ByteBuffer dst) >> 261: throws IllegalBlockSizeException, AEADBadTagException, > > Tag is generated during encryption, this can't possibly throw AEADBadTagException, copy-n-paste error maybe? Yep. copied decryptFinal. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From erikj at openjdk.java.net Thu Oct 8 15:31:04 2020 From: erikj at openjdk.java.net (Erik Joelsson) Date: Thu, 8 Oct 2020 15:31:04 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v3] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 13:59:20 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation >> (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from >> multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee >> that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class >> has been added, which defines several useful dereference routines; these are really just thin wrappers around memory >> access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not >> the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link >> to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit >> of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which >> wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as >> dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability >> in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; >> secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can >> use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done >> by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided >> below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be >> happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, >> Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd >> like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio >> Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative >> to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a >> carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access >> base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte >> offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. >> `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which >> it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both >> `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients >> can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory >> access var handle support. >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to >> achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment >> is shared, it would be possible for a thread to close it while another is accessing it. After considering several >> options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he >> reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world >> (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a >> close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and >> the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). >> It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of >> these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, >> we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to >> whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of >> stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, >> once we detect that a thread is accessing the very segment we're about to close, what should happen? We first >> experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it >> fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the >> machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to >> minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread >> is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and >> try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should >> be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single >> place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in >> addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` >> annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) >> class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, >> like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is >> tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during >> access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory >> access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead >> of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a >> `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed >> successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two >> implementations, one for confined segments and one for shared segments; the main difference between the two is what >> happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared >> segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or >> `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` >> state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### >> Memory access var handles overhaul The key realization here was that if all memory access var handles took a >> coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle >> form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var >> handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that >> e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the >> implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level >> access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see >> here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, >> since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` >> functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the >> microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared >> segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - >> https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Fix indent in GensrcScopedMemoryAccess.gmk Build changes look ok. ------------- Marked as reviewed by erikj (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/548 From ascarpino at openjdk.java.net Thu Oct 8 15:35:29 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 15:35:29 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Wed, 7 Oct 2020 20:56:06 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 1253: > >> 1251: if (decrypting) { >> 1252: if (buffered > 0) { >> 1253: cipher.decrypt(buffer, 0, buffered, new byte[0], 0); > > This looks a bit strange? The output buffer is 0-length which would lead to ShortBufferException when the buffered > bytes is enough to produce some output. This is right. decrypt() puts the data into GaloisCounterMode.ibuffer and never uses the output ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascheel at redhat.com Thu Oct 8 16:20:23 2020 From: ascheel at redhat.com (Alexander Scheel) Date: Thu, 8 Oct 2020 12:20:23 -0400 Subject: GREASE'd ALPN values - a RFC 8701 / RFC 7301 / JEP 244 discussion Message-ID: \o Hi all, I saw that ALPN support from JEP 244 was backported to JDK8 and I've recently had the time to take a closer look at it. For context, I'm one of the maintainers of JSS, a NSS wrapper for Java. I've been discussing this with another contributor, Fraser (cc'd). One of the concerns we have with the implementation (and its exposure in the corresponding SSLEngine/SSLSocket/SSLParameters interface) is that protocols are passed in as Strings. However, RFC 7301 says in section 6: > o Identification Sequence: The precise set of octet values that > identifies the protocol. This could be the UTF-8 encoding > [RFC3629] of the protocol name. When applied with GREASE'd values from RFC 8701, Strings don't work well. In particular, most of the registered values [0] are non-UTF-8, which can't be easily round-tripped in Java. This means that while precise octet values are specified by IANA, they cannot be properly specified in Java. In particular: byte[] desired = new byte[]{ (byte) 0xFA, (byte) 0xFA }; String encoded = new String(desired, StandardCharsets.UTF_8); byte[] wire = encoded.getBytes(StandardCharsets.UTF_8); String round = new String(wire, StandardCharsets.UTF_8); fails, as does choosing US_ASCII for the encoding: byte[] desired = new byte[]{ (byte) 0xFA, (byte) 0xFA }; String encoded = new String(desired, StandardCharsets.US_ASCII); byte[] wire = encoded.getBytes(StandardCharsets.UTF_8); String round = new String(wire, StandardCharsets.UTF_8); Note that we (at the application level) can't control the final (wire / round-tripped) encoding to UTF_8 as this is done within the SunJSSE implementation: https://github.com/openjdk/jdk11u-dev/blob/master/src/java.base/share/classes/sun/security/ssl/AlpnExtension.java#L100 https://github.com/openjdk/jdk11u-dev/blob/master/src/java.base/share/classes/sun/security/ssl/AlpnExtension.java#L170 https://github.com/openjdk/jdk11u-dev/blob/master/src/java.base/share/classes/sun/security/ssl/AlpnExtension.java#L223 https://github.com/openjdk/jdk11u-dev/blob/master/src/java.base/share/classes/sun/security/ssl/AlpnExtension.java#L425 and perhaps other files I'm missing. This decreases interoperability with other TLS implementations. OpenSSL [1], NSS [2], and GnuTLS [3] support setting opaque blobs as the ALPN protocol list, meaning the caller is free to supply GREASE'd values. Go on the other hand still uses its string [4], but that string class supports round-tripping non-UTF8 values correctly [5]. Additionally, it means that GREASE'd values sent by Java applications aren't compliant with the RFC 8701/IANA wire values. Is there some workaround I'm missing? I believe that setting US_ASCII internally in SunJSSE isn't sufficient to ensure the right wire encoding gets used. I'm thinking the only real fix is to deprecate the String methods and provide byte[] methods for all identifiers. Thanks in advanced, Alex [0]: https://www.iana.org/assignments/tls-extensiontype-values/tls-extensiontype-values.xhtml#alpn-protocol-ids [1]: https://www.openssl.org/docs/man1.1.0/man3/SSL_CTX_set_alpn_protos.html [2]: https://github.com/nss-dev/nss/blob/master/lib/ssl/ssl.h#L392-L409 [3]: https://gnutls.org/manual/html_node/Application-Layer-Protocol-Negotiation-_0028ALPN_0029.html [4]: https://golang.org/pkg/crypto/tls/#ClientHelloInfo [5]: https://play.golang.org/p/PjyZ-NZmKQe From mullan at openjdk.java.net Thu Oct 8 16:37:21 2020 From: mullan at openjdk.java.net (Sean Mullan) Date: Thu, 8 Oct 2020 16:37:21 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v2] In-Reply-To: References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> <-yG4ohed5byKufPfUr8_fg3W8GKVL_v0ZFNgx8o718U=.fea342a6-b204-4b84-bc13-e7837a61aed5@github.com> Message-ID: On Thu, 8 Oct 2020 14:21:09 GMT, Weijun Wang wrote: >> CSR updated. More description, and iteration counts lowered to 10000. Will update code soon. > > New commit updating ic to 10000. I also created separate constants for DEFAULT_CERT_PBE_ITERATION_COUNT and > DEFAULT_KEY_PBE_ITERATION_COUNT. I haven't made the change for LEGACY_PBE_ITERATION_COUNT since they will never change. Are you still planning, or is it possible to add a test for Windows 2019? Also, have you considered adding a test that checks if the JDK can read OpenSSL PKCS#12 files and vice versa? Maybe we can do that later as a follow-on issue. Otherwise, I will approve. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From ascarpino at openjdk.java.net Thu Oct 8 16:58:21 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 16:58:21 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Wed, 7 Oct 2020 20:56:28 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 1258: > >> 1256: } else { >> 1257: if (buffered > 0) { >> 1258: cipher.encrypt(buffer, 0, buffered, new byte[0], 0); > > Same comment as above? So there is a problem here, but not a short buffer. I hadn't realized that encrypt(byte[]..) didn't put data in the GaloisCounterMode.ibuffer because I did that with encrypt(ByteBuffer..). I added a GCM only method that will cover this and place the data in the ibuffer for doFinal(ByteBuffer..) can complete the op. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From alanb at openjdk.java.net Thu Oct 8 17:17:24 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Thu, 8 Oct 2020 17:17:24 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 11:22:09 GMT, Daniel Fuchs wrote: > Hi, > > This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of > synchronized monitors. > Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are > outlined below: > 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: > `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing > ( a snapshot of the data is taken before printing instead) > > 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: > `MeteredStream::close` was missing synchronization: it is now protected by the lock > > 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: > `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. > Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that > `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` > is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user > code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and > reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe > there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` > method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of > casting to PrintStream. > > 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: > Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. > There is no need for a synchronized here as the method is only called from a method that already > holds the lock. > > 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: > Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. > This method is only called from methods already protected by the lock. > Also the method has been made private. > > 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java > `queuedForCleanup` is made volatile as it is read and written directly from outside the class > and without protection by the `KeepAliveCleanerEntry`. > Lock protection is also added to `close()`, which was missing it. > Some methods that have no lock protection and did not need it because only called from > within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` > > 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for > `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have > acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called > from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` > > 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java > Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` > to help track the fact that AuthenticationInfo::setHeaders is only called while > holding the lock. > Synchronization was also removed from some method that didn't need it because only > called from within code blocks protected by the lock: > `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). > > 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. Mostly looks good. HttpClient.openServer - is there a reason to do the SM permission check while holding the lock? The "closed" field in MeteredStream and ChunkedInputStream needs to be volatile if you really want to read it without holding the lock. There are a couple of places with comments that I assume you added for yourself when auditing this code. There's one in HttpCapture, and 3 in HttpURLConnection. ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From ascarpino at openjdk.java.net Thu Oct 8 17:17:30 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 17:17:30 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 03:21:46 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 528: > >> 526: } >> 527: >> 528: ArrayUtil.blockSizeCheck(src.remaining(), blockSize); > > Hmm, I am not sure if this check still applies in ByteBuffer case. You are passing the ByteBuffer objs directly from > AESCipher->CipherCore->GaloisCounterMode. This is different from the byte[] case where CipherCore would chop up the > data into blocks and pass the blocks to the underlying FeedbackCipher impl. Perhaps no existing regression tests covers > ByteBuffer inputs w/ non-blocksize data? Otherwise, this should be caught? BTW, why not just use 'len' again? Seems > unnecessary to keep calling src.remaining() in various places in this method. Yes the check is unnecessary I suspect not using len was simply a mistake ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From dfuchs at openjdk.java.net Thu Oct 8 17:39:21 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Thu, 8 Oct 2020 17:39:21 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 17:14:24 GMT, Alan Bateman wrote: >> Hi, >> >> This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of >> synchronized monitors. >> Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are >> outlined below: >> 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: >> `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing >> ( a snapshot of the data is taken before printing instead) >> >> 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: >> `MeteredStream::close` was missing synchronization: it is now protected by the lock >> >> 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: >> `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. >> Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that >> `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` >> is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user >> code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and >> reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe >> there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` >> method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of >> casting to PrintStream. >> >> 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: >> Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. >> There is no need for a synchronized here as the method is only called from a method that already >> holds the lock. >> >> 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: >> Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. >> This method is only called from methods already protected by the lock. >> Also the method has been made private. >> >> 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java >> `queuedForCleanup` is made volatile as it is read and written directly from outside the class >> and without protection by the `KeepAliveCleanerEntry`. >> Lock protection is also added to `close()`, which was missing it. >> Some methods that have no lock protection and did not need it because only called from >> within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` >> >> 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for >> `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have >> acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called >> from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` >> >> 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java >> Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` >> to help track the fact that AuthenticationInfo::setHeaders is only called while >> holding the lock. >> Synchronization was also removed from some method that didn't need it because only >> called from within code blocks protected by the lock: >> `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). >> >> 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > Mostly looks good. > > HttpClient.openServer - is there a reason to do the SM permission check while holding the lock? > > The "closed" field in MeteredStream and ChunkedInputStream needs to be volatile if you really want to read it without > holding the lock. > There are a couple of places with comments that I assume you added for yourself when auditing this code. There's one in > HttpCapture, and 3 in HttpURLConnection. Thanks Alan! > HttpClient.openServer - is there a reason to do the SM permission check while holding the lock? Mostly that was a mechanical change since openServer was synchronized before. But I guess it seems also desirable for accessing host & port which are protected and not final; > The "closed" field in MeteredStream and ChunkedInputStream needs to be volatile if you really want to read it without > holding the lock. I am not so sure. `closed` starts at `false`, and can only moved from `false` to `true`. It can never go back to `false` after it's been set to `true`; So if you observe `true` outside of the lock it does means that it is really closed, isn't it? If `closed` is not volatile, the worse that can happen is that you will observe `false` while it's really `true`, and then you will need to pay the price of locking, after which you should be able to see the real `true` value. > There are a couple of places with comments that I assume you added for yourself when auditing this code. There's one in > HttpCapture, and 3 in HttpURLConnection. Yes - well - I thought that would be mostly beneficial for someone searching for `synchronized` in the java/net and sun/net packages - so that they can determine that the `synchronized` in those places was considered safe and intentionally kept unchanged, and not just overlooked. I can remove them or reformulate if you think it's better - but I was actually intending to keep these comments. ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From valeriep at openjdk.java.net Thu Oct 8 20:49:29 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Thu, 8 Oct 2020 20:49:29 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: <4a86KBug4Z58lCbvff6R2j_TOhU2tTQACh0ZcZZ7i-0=.4ca585f6-0891-4361-a796-2daf892c6ee9@github.com> On Thu, 8 Oct 2020 06:51:08 GMT, Anthony Scarpino wrote: >> 8253821: Improve ByteBuffer performance with GCM > > Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: > > Xuelei comments src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 432: > 430: // For input it takes the ibuffer which is wrapped in 'buffer' and 'src' > 431: // from doFinal. > 432: void doLastBlock(ByteBuffer buffer, ByteBuffer src, ByteBuffer dst) The ordering of these new ByteBuffer-arg methods seems random? Perhaps, either group them altogether or move them so that they are together with the byte[] counterpart methods? src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 508: > 506: } > 507: > 508: int decrypt(ByteBuffer src, ByteBuffer dst) { Similar to above. It seems a bit strange to see decrypt(...) method in between encrypt(...) methods. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From coleenp at openjdk.java.net Thu Oct 8 20:53:29 2020 From: coleenp at openjdk.java.net (Coleen Phillimore) Date: Thu, 8 Oct 2020 20:53:29 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v3] In-Reply-To: References: Message-ID: <04N7NDBB9K0OIRpJy4FaS7jPns7J65f5vpIPAKdLdIQ=.d72b8f73-6062-4bbf-a9c3-64d41de9165d@github.com> On Thu, 8 Oct 2020 13:59:20 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation >> (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from >> multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee >> that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class >> has been added, which defines several useful dereference routines; these are really just thin wrappers around memory >> access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not >> the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link >> to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit >> of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which >> wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as >> dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability >> in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; >> secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can >> use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done >> by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided >> below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be >> happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, >> Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd >> like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio >> Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative >> to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a >> carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access >> base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte >> offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. >> `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which >> it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both >> `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients >> can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory >> access var handle support. >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to >> achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment >> is shared, it would be possible for a thread to close it while another is accessing it. After considering several >> options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he >> reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world >> (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a >> close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and >> the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). >> It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of >> these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, >> we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to >> whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of >> stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, >> once we detect that a thread is accessing the very segment we're about to close, what should happen? We first >> experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it >> fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the >> machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to >> minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread >> is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and >> try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should >> be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single >> place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in >> addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` >> annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) >> class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, >> like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is >> tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during >> access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory >> access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead >> of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a >> `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed >> successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two >> implementations, one for confined segments and one for shared segments; the main difference between the two is what >> happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared >> segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or >> `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` >> state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### >> Memory access var handles overhaul The key realization here was that if all memory access var handles took a >> coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle >> form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var >> handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that >> e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the >> implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level >> access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see >> here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, >> since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` >> functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the >> microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared >> segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - >> https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Fix indent in GensrcScopedMemoryAccess.gmk just a drive-by comment. src/hotspot/share/classfile/vmSymbols.hpp line 290: > 288: template(jdk_internal_vm_annotation_ForceInline_signature, "Ljdk/internal/vm/annotation/ForceInline;") \ > 289: template(jdk_internal_vm_annotation_Hidden_signature, "Ljdk/internal/vm/annotation/Hidden;") \ > 290: template(jdk_internal_misc_Scoped_signature, "Ljdk/internal/misc/ScopedMemoryAccess$Scoped;") \ Can you line this up? ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From ascarpino at openjdk.java.net Thu Oct 8 22:04:26 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 8 Oct 2020 22:04:26 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 01:08:44 GMT, Anthony Scarpino wrote: >> src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 658: >> >>> 656: BadPaddingException { >>> 657: return bufferCrypt(input, output, false); >>> 658: } >> >> Is the override of this method for using a different bufferCrypt impl? There is also engineUpdate(ByteBuffer, >> ByteBuffer) in CipherSpi, is there a reason for not overriding that here? > > Yes. thanks.. The IDE covered that up by going to the one in CipherSpi and I thought it was calling the AES one. TLS > doesn't use update() so the perf numbers won't change. I'll have to run the tests again. I'm not going to do the update() method, leaving it as is. There is some complications with the Encrypt.java test were the update is done with a direct bytebuffer, but the doFinal() is an empty buffer, which sends it to the byte[] doFinal(). CipherCore mitigates this situation inefficiently and I'd rather optimize that in future changeset that I'm already planning for byte[] methods. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From psandoz at openjdk.java.net Thu Oct 8 23:20:22 2020 From: psandoz at openjdk.java.net (Paul Sandoz) Date: Thu, 8 Oct 2020 23:20:22 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v3] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 13:59:20 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation >> (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from >> multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee >> that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class >> has been added, which defines several useful dereference routines; these are really just thin wrappers around memory >> access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not >> the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link >> to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit >> of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which >> wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as >> dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability >> in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; >> secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can >> use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done >> by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided >> below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be >> happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, >> Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd >> like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio >> Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative >> to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a >> carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access >> base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte >> offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. >> `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which >> it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both >> `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients >> can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory >> access var handle support. >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to >> achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment >> is shared, it would be possible for a thread to close it while another is accessing it. After considering several >> options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he >> reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world >> (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a >> close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and >> the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). >> It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of >> these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, >> we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to >> whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of >> stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, >> once we detect that a thread is accessing the very segment we're about to close, what should happen? We first >> experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it >> fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the >> machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to >> minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread >> is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and >> try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should >> be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single >> place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in >> addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` >> annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) >> class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, >> like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is >> tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during >> access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory >> access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead >> of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a >> `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed >> successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two >> implementations, one for confined segments and one for shared segments; the main difference between the two is what >> happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared >> segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or >> `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` >> state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### >> Memory access var handles overhaul The key realization here was that if all memory access var handles took a >> coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle >> form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var >> handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that >> e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the >> implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level >> access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see >> here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, >> since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` >> functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the >> microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared >> segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - >> https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Fix indent in GensrcScopedMemoryAccess.gmk Reviewed this when updated in [panama-foreign](https://github.com/openjdk/panama-foreign/tree/foreign-memaccess), hence the lack of substantial comments for this PR. src/java.base/share/classes/jdk/internal/misc/X-ScopedMemoryAccess.java.template line 50: > 48: * a memory region while another thread is releasing it. > 49: *

> 50: * This class provides tools to manages races when multiple threads are accessing and/or releasing the same memory s/manages/manage test/jdk/java/foreign/TestCleaner.java line 183: > 181: for (int cleaner = 0 ; cleaner < cleaners.length ; cleaner++) { > 182: for (int segmentFunction = 0 ; segmentFunction < segmentFunctions.length ; segmentFunction++) { > 183: data[kind + kinds.length * cleaner + (cleaners.length * kinds.length * segmentFunction)] = Using an `ArrayList` with `list.toArray(Object[][]::new)` would make this easier to read: List data = new ArrayList<>(); for (Registered kind : RegisterKind.values()) { for (Object cleaner : cleaners) { for (SegmentFunction segmentFunction : SegmentFunction.values()) { data.add(new Object[] {kind, cleaner, segmentFunction}); } } } return data.toArray(Object[][]::new); ------------- Marked as reviewed by psandoz (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/548 From valeriep at openjdk.java.net Thu Oct 8 23:23:24 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Thu, 8 Oct 2020 23:23:24 GMT Subject: RFR: 8253563: Change sun.security.jca.Providers.threadLists to be ThreadLocal Message-ID: Could someone help reviewing this one-line change? This changes the provider list used during jar verification from InheritableThreadLocal to ThreadLocal. Existing usage and handling uses this field as temporary thread-specific provider list for jar verification. There seems to be no reason for it to be InheritableThreadLocal. Existing regression tests pass with this change. Thanks, Valerie ------------- Commit messages: - 8253563: Change sun.security.jca.Providers.threadLists to be ThreadLocal Changes: https://git.openjdk.java.net/jdk/pull/570/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=570&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8253563 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/570.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/570/head:pull/570 PR: https://git.openjdk.java.net/jdk/pull/570 From weijun at openjdk.java.net Fri Oct 9 00:07:21 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 9 Oct 2020 00:07:21 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v2] In-Reply-To: References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> <-yG4ohed5byKufPfUr8_fg3W8GKVL_v0ZFNgx8o718U=.fea342a6-b204-4b84-bc13-e7837a61aed5@github.com> Message-ID: <3JzBIcRF0FdFs0y3wY61scaqfgpYSqOa3jrRCP67Fv4=.6a3c3e98-22d9-4ef0-8c96-f04074c10f2e@github.com> On Thu, 8 Oct 2020 16:34:59 GMT, Sean Mullan wrote: >> New commit updating ic to 10000. I also created separate constants for DEFAULT_CERT_PBE_ITERATION_COUNT and >> DEFAULT_KEY_PBE_ITERATION_COUNT. I haven't made the change for LEGACY_PBE_ITERATION_COUNT since they will never change. > > Are you still planning, or is it possible to add a test for Windows 2019? Also, have you considered adding a test that > checks if the JDK can read OpenSSL PKCS#12 files and vice versa? Maybe we can do that later as a follow-on issue. > Otherwise, I will approve. I tried but cannot find a way to tell if a system is Windows Server 2016 or 2019. Their os.version is all 10.0. I've filed an enhancement for it. That said, I did try running the test using new algorithms and it succeeds. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Fri Oct 9 00:10:22 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 9 Oct 2020 00:10:22 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v2] In-Reply-To: <3JzBIcRF0FdFs0y3wY61scaqfgpYSqOa3jrRCP67Fv4=.6a3c3e98-22d9-4ef0-8c96-f04074c10f2e@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> <-yG4ohed5byKufPfUr8_fg3W8GKVL_v0ZFNgx8o718U=.fea342a6-b204-4b84-bc13-e7837a61aed5@github.com> <3JzBIcRF0FdFs0y3wY61scaqfgpYSqOa3jrRCP67Fv4=.6a3c3e98-22d9-4ef0-8c96-f04074c10f2e@github.com> Message-ID: <5FQISOrJ6h-Os7JByIV_pqmhLLDDdWfwAtFoeqhT0RA=.953d53e5-7d99-4234-8fa3-bc22d4ee588c@github.com> On Fri, 9 Oct 2020 00:04:17 GMT, Weijun Wang wrote: >> Are you still planning, or is it possible to add a test for Windows 2019? Also, have you considered adding a test that >> checks if the JDK can read OpenSSL PKCS#12 files and vice versa? Maybe we can do that later as a follow-on issue. >> Otherwise, I will approve. > > I tried but cannot find a way to tell if a system is Windows Server 2016 or 2019. Their os.version is all 10.0. I've > filed an enhancement at https://bugs.openjdk.java.net/browse/JDK-8254241 for it. That said, I did try running the test > on a Windows Server 2019 using new algorithms and it succeeds. There are existing tests reading openssl generated pkcs12 files in https://github.com/openjdk/jdk/tree/master/test/jdk/sun/security/pkcs12/params, but I can add a new one using strong algorithms. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From valeriep at openjdk.java.net Fri Oct 9 00:45:27 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 9 Oct 2020 00:45:27 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 06:51:08 GMT, Anthony Scarpino wrote: >> 8253821: Improve ByteBuffer performance with GCM > > Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: > > Xuelei comments src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 545: > 543: "Unable to add remaining input to the buffer", e); > 544: } > 545: } Existing usage for ibuffer is only for decryption. If you are storing into ibuffer for encryption, the comment about it should be updated. The earlier code in this method should also check ibuffer and if non-empty, apply its content before applying src? ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From weijun at openjdk.java.net Fri Oct 9 01:33:38 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 9 Oct 2020 01:33:38 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v3] In-Reply-To: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: > Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at > https://bugs.openjdk.java.net/browse/JDK-8228481. Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: update README and exclude README ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/473/files - new: https://git.openjdk.java.net/jdk/pull/473/files/6b5c5b5e..41be78aa Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=473&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=473&range=01-02 Stats: 39 lines in 2 files changed: 7 ins; 6 del; 26 mod Patch: https://git.openjdk.java.net/jdk/pull/473.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/473/head:pull/473 PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Fri Oct 9 01:33:38 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 9 Oct 2020 01:33:38 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v3] In-Reply-To: <5FQISOrJ6h-Os7JByIV_pqmhLLDDdWfwAtFoeqhT0RA=.953d53e5-7d99-4234-8fa3-bc22d4ee588c@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> <2aTk_UQXvFl5rW4FHat0-gvkig-iF018ddcBt8lTH6w=.29f0e0da-c65c-4f5e-bab2-e1a821459a8b@github.com> <-yG4ohed5byKufPfUr8_fg3W8GKVL_v0ZFNgx8o718U=.fea342a6-b204-4b84-bc13-e7837a61aed5@github.com> <3JzBIcRF0FdFs0y3wY61scaqfgpYSqOa3jrRCP67Fv4=.6a3c3e98-22d9-4ef0-8c96-f04074c10f2e@github.com> <5FQISOrJ6h-Os7JByIV_pqmhLLDDdWfwAtFoeqhT0RA=.953d53e5-7d99-4234-8fa3-bc22d4ee588c@github.com> Message-ID: <3RlNMNwIqZFJNuJ5wzeKPW4AkUWkXKbHV6S8Qw0BOgc=.b2fc2252-c24d-465c-afdc-b4d84251bab6@github.com> On Fri, 9 Oct 2020 00:07:39 GMT, Weijun Wang wrote: >> I tried but cannot find a way to tell if a system is Windows Server 2016 or 2019. Their os.version is all 10.0. I've >> filed an enhancement at https://bugs.openjdk.java.net/browse/JDK-8254241 for it. That said, I did try running the test >> on a Windows Server 2019 using new algorithms and it succeeds. > > There are existing tests reading openssl generated pkcs12 files in > https://github.com/openjdk/jdk/tree/master/test/jdk/sun/security/pkcs12/params, it already contains files using both > weak and strong algorithms. Update `params/README`, exclude it from the de-BASE64 list (don't know it succeeded) in the `ParamsTest.java` test. Also remove a useless call in the test. Thinking about adding a benchmark, but it will be in another commit. ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From ascarpino at openjdk.java.net Fri Oct 9 01:55:25 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Fri, 9 Oct 2020 01:55:25 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Fri, 9 Oct 2020 00:42:57 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 545: > >> 543: "Unable to add remaining input to the buffer", e); >> 544: } >> 545: } > > Existing usage for ibuffer is only for decryption. If you are storing into ibuffer for encryption, the comment about it > should be updated. The earlier code in this method should also check ibuffer and if non-empty, apply its content before > applying src? I updated the comment. I had redone this method before your comment. Take a look at that on the next webrev ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From valeriep at openjdk.java.net Fri Oct 9 04:01:23 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 9 Oct 2020 04:01:23 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> On Thu, 8 Oct 2020 06:51:08 GMT, Anthony Scarpino wrote: >> 8253821: Improve ByteBuffer performance with GCM > > Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: > > Xuelei comments src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 550: > 548: > 549: processed += len; > 550: ghashAllToS.update(src, len); Isn't input to ghashAllToS always be the produced cipher text? Did I miss something? src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 595: > 593: } > 594: GCTR gctrForSToTag = new GCTR(embeddedCipher, this.preCounterBlock); > 595: gctrForSToTag.doFinal(s, 0, s.length, block, 0); since GCTR output the same length of output as input, (e.g. 'sOut' is same length as 's'), can't we just re-use 's' as the output buffer instead of 'block'? src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 593: > 591: if (tagLenBytes > block.length) { > 592: block = new byte[tagLenBytes]; > 593: } Is tagLenBytes ever larger than AES block size? src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 743: > 741: } > 742: GCTR gctrForSToTag = new GCTR(embeddedCipher, this.preCounterBlock); > 743: gctrForSToTag.doFinal(s, 0, tagLenBytes, block, 0); Same comments as earlier. src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 748: > 746: int mismatch = 0; > 747: for (int i = 0; i < tagLenBytes; i++) { > 748: mismatch |= tag[i] ^ block[i]; block->s? src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 605: > 603: int len = src.remaining(); > 604: dst.mark(); > 605: if (len > MAX_BUF_SIZE - tagLenBytes) { Missing the bytes in ibuffer for this check? src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 610: > 608: } > 609: > 610: if (dst.remaining() < len + tagLenBytes) { Missing the bytes in ibuffer for this check? src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 614: > 612: } > 613: > 614: checkDataLength(processed, len); It seems that both checks (line 605 and 614) can be combined into: checkDataLength(processed, Math.addExact(len, tagLenBytes)); src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 630: > 628: if (tagLenBytes > block.length) { > 629: block = new byte[tagLenBytes]; > 630: } Again, will this ever happen? Can just use 's' instead of 'block' from here and below? src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 635: > 633: dst.put(block, 0, tagLenBytes); > 634: > 635: return (processed + tagLenBytes); Is it supposed to return "all data processed + tag len"? Normally, we return the output size produced by this particular call instead of all accumulated. src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 617: > 615: > 616: processAAD(); > 617: if (len > 0) { Even if (len == 0), we should still process the data stored into 'ibuffer'? It seems that both of the encrypt(ByteBuffer) and encryptFinal(ByteBuffer) are adapted from their counterpart with byte[] arguments. However, the byte[] methods have different entrant conditions due to the buffering in CipherCore. So the impl of the ByteBuffer ones may need additional logic to handle all possible calling sequence. src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 621: > 619: null : ByteBuffer.wrap(ibuffer.toByteArray()), src, dst); > 620: dst.reset(); > 621: ghashAllToS.doLastBlock(dst, processed); Are we sure about using "processed" here? I'd expect the value is the number of bytes written into dst in the doLastBlock(...) call on line 618. Is the "processed" variable used differently in ByteBuffer case? ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Fri Oct 9 04:29:20 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Fri, 9 Oct 2020 04:29:20 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> References: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> Message-ID: On Fri, 9 Oct 2020 01:04:26 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 593: > >> 591: if (tagLenBytes > block.length) { >> 592: block = new byte[tagLenBytes]; >> 593: } > > Is tagLenBytes ever larger than AES block size? no, 16 octets ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Fri Oct 9 04:41:20 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Fri, 9 Oct 2020 04:41:20 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> References: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> Message-ID: On Fri, 9 Oct 2020 01:01:47 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 595: > >> 593: } >> 594: GCTR gctrForSToTag = new GCTR(embeddedCipher, this.preCounterBlock); >> 595: gctrForSToTag.doFinal(s, 0, s.length, block, 0); > > since GCTR output the same length of output as input, (e.g. 'sOut' is same length as 's'), can't we just re-use 's' as > the output buffer instead of 'block'? Actually I can take it one step further.. I think I can remove 's' and use 'block' for everything. I'll have to make sure no unexpected overwriting happens. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Fri Oct 9 04:46:23 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Fri, 9 Oct 2020 04:46:23 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> References: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> Message-ID: On Fri, 9 Oct 2020 03:20:40 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 635: > >> 633: dst.put(block, 0, tagLenBytes); >> 634: >> 635: return (processed + tagLenBytes); > > Is it supposed to return "all data processed + tag len"? Normally, we return the output size produced by this > particular call instead of all accumulated. Yes, that should be len + tagLenBytes. We obviously don't check that in our tests ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Fri Oct 9 05:01:21 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Fri, 9 Oct 2020 05:01:21 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> References: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> Message-ID: On Fri, 9 Oct 2020 00:48:42 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 550: > >> 548: >> 549: processed += len; >> 550: ghashAllToS.update(src, len); > > Isn't input to ghashAllToS always be the produced cipher text? Did I miss something? method is removed > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 617: > >> 615: >> 616: processAAD(); >> 617: if (len > 0) { > > Even if (len == 0), we should still process the data stored into 'ibuffer'? It seems that both of the > encrypt(ByteBuffer) and encryptFinal(ByteBuffer) are adapted from their counterpart with byte[] arguments. However, the > byte[] methods have different entrant conditions due to the buffering in CipherCore. So the impl of the ByteBuffer ones > may need additional logic to handle all possible calling sequence. Yes for encryptFinal. encrypt is removed ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Fri Oct 9 05:05:21 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Fri, 9 Oct 2020 05:05:21 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> References: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> Message-ID: On Fri, 9 Oct 2020 03:58:38 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 621: > >> 619: null : ByteBuffer.wrap(ibuffer.toByteArray()), src, dst); >> 620: dst.reset(); >> 621: ghashAllToS.doLastBlock(dst, processed); > > Are we sure about using "processed" here? I'd expect the value is the number of bytes written into dst in the > doLastBlock(...) call on line 618. Is the "processed" variable used differently in ByteBuffer case? it should probably be 'len' ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From alanb at openjdk.java.net Fri Oct 9 07:08:27 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 9 Oct 2020 07:08:27 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 17:36:31 GMT, Daniel Fuchs wrote: > Mostly that was a mechanical change since openServer was synchronized before. > But I guess it seems also desirable for accessing host & port which are protected and not final; Okay, I mis-read the old code. The syncrhonization to access host/port isn't obvious so leaving it as is is okay. > I am not so sure. `closed` starts at `false`, and can only moved from `false` to `true`. It can never go back to > `false` after it's been set to `true`; So if you observe `true` outside of the lock it does means that it is really > closed, isn't it? If `closed` is not volatile, the worse that can happen is that you will observe `false` while it's > really `true`, and then you will need to pay the price of locking, after which you should be able to see the real > `true` value. Allowing for visibility failures here is confusing for maintainers. If you really want to access closed without the lock (and I don't see any reason to do that) then it would be clearer to all if it were volatile. > Yes - well - I thought that would be mostly beneficial for someone searching for `synchronized` in the java/net and > sun/net packages - so that they can determine that the `synchronized` in those places was considered safe and > intentionally kept unchanged, and not just overlooked. I can remove them or reformulate if you think it's better - but > I was actually intending to keep these comments. I don't have. a strong opinion here but they did initially look like left over comments. Will they mean anything to someone looking at this code in 2025? ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From chegar at openjdk.java.net Fri Oct 9 09:04:22 2020 From: chegar at openjdk.java.net (Chris Hegarty) Date: Fri, 9 Oct 2020 09:04:22 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 11:22:09 GMT, Daniel Fuchs wrote: > Hi, > > This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of > synchronized monitors. > Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are > outlined below: > 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: > `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing > ( a snapshot of the data is taken before printing instead) > > 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: > `MeteredStream::close` was missing synchronization: it is now protected by the lock > > 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: > `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. > Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that > `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` > is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user > code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and > reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe > there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` > method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of > casting to PrintStream. > > 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: > Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. > There is no need for a synchronized here as the method is only called from a method that already > holds the lock. > > 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: > Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. > This method is only called from methods already protected by the lock. > Also the method has been made private. > > 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java > `queuedForCleanup` is made volatile as it is read and written directly from outside the class > and without protection by the `KeepAliveCleanerEntry`. > Lock protection is also added to `close()`, which was missing it. > Some methods that have no lock protection and did not need it because only called from > within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` > > 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for > `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have > acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called > from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` > > 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java > Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` > to help track the fact that AuthenticationInfo::setHeaders is only called while > holding the lock. > Synchronization was also removed from some method that didn't need it because only > called from within code blocks protected by the lock: > `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). > > 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. Mostly looks good. Just a few comments. src/java.base/share/classes/sun/net/www/MessageHeader.java line 330: > 328: at the end. Omits pairs with a null key. Omits > 329: colon if key-value pair is the requestline. */ > 330: private void print(int nkeys, String[] keys, String[] values, PrintStream p) { While not strictly necessary, this method (along with isRequestline) could be _static_. Which ensures that their implementations do not access instance fields. src/java.base/share/classes/sun/net/www/MeteredStream.java line 123: > 121: lock(); > 122: try { > 123: if (closed) return -1; This double check of `closed` is kind of irritating. Is it really need, or maybe we just drop it and lock unconditionally? src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java line 69: > 67: public void close() throws IOException { > 68: // If the inputstream is closed already, or if this stream > 69: // has already been queued for cleanup.just return. Minor typo, "cleanup.just" ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From dfuchs at openjdk.java.net Fri Oct 9 09:20:18 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Fri, 9 Oct 2020 09:20:18 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers In-Reply-To: References: Message-ID: <5ozPffL2Nf9a84cIlodNQskaGWTab98tgpEPRgPOMkQ=.2d343a27-a6f0-415d-b80e-570b790241f3@github.com> On Fri, 9 Oct 2020 08:36:03 GMT, Chris Hegarty wrote: >> Hi, >> >> This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of >> synchronized monitors. >> Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are >> outlined below: >> 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: >> `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing >> ( a snapshot of the data is taken before printing instead) >> >> 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: >> `MeteredStream::close` was missing synchronization: it is now protected by the lock >> >> 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: >> `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. >> Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that >> `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` >> is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user >> code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and >> reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe >> there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` >> method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of >> casting to PrintStream. >> >> 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: >> Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. >> There is no need for a synchronized here as the method is only called from a method that already >> holds the lock. >> >> 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: >> Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. >> This method is only called from methods already protected by the lock. >> Also the method has been made private. >> >> 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java >> `queuedForCleanup` is made volatile as it is read and written directly from outside the class >> and without protection by the `KeepAliveCleanerEntry`. >> Lock protection is also added to `close()`, which was missing it. >> Some methods that have no lock protection and did not need it because only called from >> within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` >> >> 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for >> `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have >> acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called >> from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` >> >> 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java >> Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` >> to help track the fact that AuthenticationInfo::setHeaders is only called while >> holding the lock. >> Synchronization was also removed from some method that didn't need it because only >> called from within code blocks protected by the lock: >> `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). >> >> 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > src/java.base/share/classes/sun/net/www/MeteredStream.java line 123: > >> 121: lock(); >> 122: try { >> 123: if (closed) return -1; > > This double check of `closed` is kind of irritating. Is it really need, or maybe we just drop it and lock > unconditionally? We could. ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From simonis at openjdk.java.net Fri Oct 9 10:28:34 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Fri, 9 Oct 2020 10:28:34 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v2] In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/520/files - new: https://git.openjdk.java.net/jdk/pull/520/files/149b7054..2a9427ef Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=00-01 Stats: 26 lines in 2 files changed: 0 ins; 20 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/520.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/520/head:pull/520 PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Fri Oct 9 10:28:34 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Fri, 9 Oct 2020 10:28:34 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Wed, 7 Oct 2020 16:45:53 GMT, Volker Simonis wrote: >> I think as we start to move forward with the review and CSR, we should update the bug and PR description as the change >> is not a workaround but a change in behavior of the implementation > > I already changed the bug description on Alan's request and just updated the PR description to reflect that. Please > feel free to propose a better description. Thanks for your feedback. I've updated the API doc as requested and created a CSR at: https://bugs.openjdk.java.net/browse/JDK-8254284 Can somebody please also review the CSR? Thank you and best regards, Volker ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From mcimadamore at openjdk.java.net Fri Oct 9 10:39:56 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Fri, 9 Oct 2020 10:39:56 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v4] In-Reply-To: References: Message-ID: <4h3RYe5xrN6HpjRM1kkVgfo9yDiPoxoS312x9nYDpwA=.3260dd59-63e3-4b55-8a86-d512b1a4ee24@github.com> > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Address review comments ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/b941c4a2..d96c32ac Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=02-03 Stats: 16 lines in 3 files changed: 2 ins; 6 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From michaelm at openjdk.java.net Fri Oct 9 10:49:20 2020 From: michaelm at openjdk.java.net (Michael McMahon) Date: Fri, 9 Oct 2020 10:49:20 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers In-Reply-To: <5ozPffL2Nf9a84cIlodNQskaGWTab98tgpEPRgPOMkQ=.2d343a27-a6f0-415d-b80e-570b790241f3@github.com> References: <5ozPffL2Nf9a84cIlodNQskaGWTab98tgpEPRgPOMkQ=.2d343a27-a6f0-415d-b80e-570b790241f3@github.com> Message-ID: On Fri, 9 Oct 2020 09:17:48 GMT, Daniel Fuchs wrote: >> src/java.base/share/classes/sun/net/www/MeteredStream.java line 123: >> >>> 121: lock(); >>> 122: try { >>> 123: if (closed) return -1; >> >> This double check of `closed` is kind of irritating. Is it really need, or maybe we just drop it and lock >> unconditionally? > > We could. That's a good suggestion. I agree on that point. ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From mcimadamore at openjdk.java.net Fri Oct 9 11:34:56 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Fri, 9 Oct 2020 11:34:56 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v5] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Fix performance issue with "small" segment mismatch ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/d96c32ac..9b3fc227 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=03-04 Stats: 7 lines in 1 file changed: 0 ins; 5 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Fri Oct 9 11:37:19 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Fri, 9 Oct 2020 11:37:19 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v3] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 23:17:33 GMT, Paul Sandoz wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Fix indent in GensrcScopedMemoryAccess.gmk > > Reviewed this when updated in [panama-foreign](https://github.com/openjdk/panama-foreign/tree/foreign-memaccess), hence > the lack of substantial comments for this PR. When re-running all benchmarks, I noted an issue with the `BulkOps` microbenchmark: calling `MemorySegment::mismatch` on a small segment (< 8 bytes) was 10x slower than with ByteBuffers. After some investigation, I realized that the issue is caused by the fact that the `AbstractMemorySegmentImpl::mismatch` method contains inexact var handle calls, where the segment coordinate has type `AbstractMemorySegmentImpl` instead of the expected `MemorySegment`, so we take the slow path. A simple solution is to avoid using var handles directly here, and use the helper functions in MemoryAccess, which widens the type accordingly and produce an exact var handle call. With this change, perfomance of mismatch on small segment is on par with ByteBuffer. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From dfuchs at openjdk.java.net Fri Oct 9 11:49:30 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Fri, 9 Oct 2020 11:49:30 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v2] In-Reply-To: References: Message-ID: > Hi, > > This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of > synchronized monitors. > Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are > outlined below: > 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: > `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing > ( a snapshot of the data is taken before printing instead) > > 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: > `MeteredStream::close` was missing synchronization: it is now protected by the lock > > 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: > `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. > Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that > `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` > is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user > code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and > reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe > there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` > method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of > casting to PrintStream. > > 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: > Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. > There is no need for a synchronized here as the method is only called from a method that already > holds the lock. > > 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: > Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. > This method is only called from methods already protected by the lock. > Also the method has been made private. > > 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java > `queuedForCleanup` is made volatile as it is read and written directly from outside the class > and without protection by the `KeepAliveCleanerEntry`. > Lock protection is also added to `close()`, which was missing it. > Some methods that have no lock protection and did not need it because only called from > within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` > > 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for > `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have > acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called > from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` > > 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java > Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` > to help track the fact that AuthenticationInfo::setHeaders is only called while > holding the lock. > Synchronization was also removed from some method that didn't need it because only > called from within code blocks protected by the lock: > `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). > > 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. Daniel Fuchs has updated the pull request incrementally with one additional commit since the last revision: 8229867: Re-examine synchronization usages in http and https protocol handlers Incorporated review feedback ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/558/files - new: https://git.openjdk.java.net/jdk/pull/558/files/c8dc2ac9..ddfa2e6c Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=558&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=558&range=00-01 Stats: 35 lines in 8 files changed: 15 ins; 14 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/558.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/558/head:pull/558 PR: https://git.openjdk.java.net/jdk/pull/558 From fyang at openjdk.java.net Fri Oct 9 11:53:31 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Fri, 9 Oct 2020 11:53:31 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v5] In-Reply-To: References: Message-ID: > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. > Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 > > Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. > For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. > "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. > > We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. > Patch passed jtreg tier1-3 tests with QEMU system emulator. > Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make > sure that there's no regression. > We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java > We measured the performance benefit with an aarch64 cycle-accurate simulator. > Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. > > For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on > real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is > detected. Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains six commits: - Merge master - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check - Rebase - Merge master - Fix trailing whitespace issue - 8252204: AArch64: Implement SHA3 accelerator/intrinsic Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com ------------- Changes: https://git.openjdk.java.net/jdk/pull/207/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=207&range=04 Stats: 1512 lines in 35 files changed: 1025 ins; 22 del; 465 mod Patch: https://git.openjdk.java.net/jdk/pull/207.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/207/head:pull/207 PR: https://git.openjdk.java.net/jdk/pull/207 From dfuchs at openjdk.java.net Fri Oct 9 12:08:44 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Fri, 9 Oct 2020 12:08:44 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v2] In-Reply-To: References: Message-ID: On Fri, 9 Oct 2020 08:49:54 GMT, Chris Hegarty wrote: >> Daniel Fuchs has updated the pull request incrementally with one additional commit since the last revision: >> >> 8229867: Re-examine synchronization usages in http and https protocol handlers >> >> Incorporated review feedback > > src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java line 69: > >> 67: public void close() throws IOException { >> 68: // If the inputstream is closed already, or if this stream >> 69: // has already been queued for cleanup.just return. > > Minor typo, "cleanup.just" done > src/java.base/share/classes/sun/net/www/MessageHeader.java line 330: > >> 328: at the end. Omits pairs with a null key. Omits >> 329: colon if key-value pair is the requestline. */ >> 330: private void print(int nkeys, String[] keys, String[] values, PrintStream p) { > > While not strictly necessary, this method (along with isRequestline) could be _static_. Which ensures that their > implementations do not access instance fields. Done. ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From dfuchs at openjdk.java.net Fri Oct 9 12:12:05 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Fri, 9 Oct 2020 12:12:05 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v2] In-Reply-To: References: Message-ID: On Fri, 9 Oct 2020 09:01:43 GMT, Chris Hegarty wrote: >> Daniel Fuchs has updated the pull request incrementally with one additional commit since the last revision: >> >> 8229867: Re-examine synchronization usages in http and https protocol handlers >> >> Incorporated review feedback > > Mostly looks good. Just a few comments. >From Alan: > Allowing for visibility failures here is confusing for maintainers. If you really want to access closed without the > lock (and I don't see any reason to do that) then it would be clearer to all if it were volatile. >From Chris: > This double check of closed is kind of irritating. Is it really need, or maybe we just drop it and lock unconditionally? => I have removed the double locking. >From Alan: > I don't have a strong opinion here but they did initially look like left over comments. Will they mean anything to > someone looking at this code in 2025? I have updated the comments to be more explanatory for future maintainer (especially if they use the Annotate feature of the IDE to correlate with the fix in which they were introduced). That said, if you prefer removing them altogether let me know, and I'll remove them. ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From aph at openjdk.java.net Fri Oct 9 12:22:15 2020 From: aph at openjdk.java.net (Andrew Haley) Date: Fri, 9 Oct 2020 12:22:15 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v5] In-Reply-To: References: Message-ID: On Fri, 9 Oct 2020 11:53:31 GMT, Fei Yang wrote: >> Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com >> >> This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. >> Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 >> >> Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. >> For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. >> "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. >> >> We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. >> Patch passed jtreg tier1-3 tests with QEMU system emulator. >> Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make >> sure that there's no regression. >> We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java >> We measured the performance benefit with an aarch64 cycle-accurate simulator. >> Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. >> >> For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on >> real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is >> detected. > > Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains > six commits: > - Merge master > - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check > - Rebase > - Merge master > - Fix trailing whitespace issue > - 8252204: AArch64: Implement SHA3 accelerator/intrinsic > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com Marked as reviewed by aph (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From alanb at openjdk.java.net Fri Oct 9 13:25:10 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 9 Oct 2020 13:25:10 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v2] In-Reply-To: References: Message-ID: On Fri, 9 Oct 2020 11:49:30 GMT, Daniel Fuchs wrote: >> Hi, >> >> This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of >> synchronized monitors. >> Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are >> outlined below: >> 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: >> `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing >> ( a snapshot of the data is taken before printing instead) >> >> 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: >> `MeteredStream::close` was missing synchronization: it is now protected by the lock >> >> 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: >> `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. >> Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that >> `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` >> is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user >> code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and >> reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe >> there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` >> method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of >> casting to PrintStream. >> >> 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: >> Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. >> There is no need for a synchronized here as the method is only called from a method that already >> holds the lock. >> >> 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: >> Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. >> This method is only called from methods already protected by the lock. >> Also the method has been made private. >> >> 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java >> `queuedForCleanup` is made volatile as it is read and written directly from outside the class >> and without protection by the `KeepAliveCleanerEntry`. >> Lock protection is also added to `close()`, which was missing it. >> Some methods that have no lock protection and did not need it because only called from >> within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` >> >> 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for >> `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have >> acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called >> from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` >> >> 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java >> Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` >> to help track the fact that AuthenticationInfo::setHeaders is only called while >> holding the lock. >> Synchronization was also removed from some method that didn't need it because only >> called from within code blocks protected by the lock: >> `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). >> >> 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > Daniel Fuchs has updated the pull request incrementally with one additional commit since the last revision: > > 8229867: Re-examine synchronization usages in http and https protocol handlers > > Incorporated review feedback src/java.base/share/classes/sun/net/www/http/HttpCapture.java line 59: > 57: // Although accessing files could result in blocking operations, > 58: // HttpCapture is a corner case; there seem no urgent need to convert > 59: // this class to using java.util.concurrent.locks at this time. The updated patch looks good but I think this comment needs another iteration to ensure that it doesn't confuse future maintainers. You could drop it or else replace it with something simple that says that HttpCapture does blocking I/O operations while holding monitors but it's not a concern because it rarely used. ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From psandoz at openjdk.java.net Fri Oct 9 15:26:11 2020 From: psandoz at openjdk.java.net (Paul Sandoz) Date: Fri, 9 Oct 2020 15:26:11 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v5] In-Reply-To: References: Message-ID: On Fri, 9 Oct 2020 11:34:56 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation >> (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from >> multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee >> that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class >> has been added, which defines several useful dereference routines; these are really just thin wrappers around memory >> access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not >> the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link >> to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit >> of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which >> wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as >> dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability >> in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; >> secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can >> use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done >> by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided >> below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be >> happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, >> Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd >> like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio >> Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative >> to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a >> carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access >> base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte >> offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. >> `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which >> it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both >> `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients >> can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory >> access var handle support. >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to >> achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment >> is shared, it would be possible for a thread to close it while another is accessing it. After considering several >> options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he >> reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world >> (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a >> close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and >> the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). >> It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of >> these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, >> we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to >> whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of >> stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, >> once we detect that a thread is accessing the very segment we're about to close, what should happen? We first >> experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it >> fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the >> machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to >> minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread >> is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and >> try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should >> be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single >> place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in >> addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` >> annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) >> class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, >> like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is >> tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during >> access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory >> access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead >> of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a >> `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed >> successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two >> implementations, one for confined segments and one for shared segments; the main difference between the two is what >> happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared >> segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or >> `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` >> state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### >> Memory access var handles overhaul The key realization here was that if all memory access var handles took a >> coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle >> form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var >> handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that >> e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the >> implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level >> access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see >> here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, >> since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` >> functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the >> microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared >> segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - >> https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Fix performance issue with "small" segment mismatch src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/Utils.java line 76: > 74: // This adaptation is required, otherwise the memory access var handle will have type MemoryAddressProxy, > 75: // and not MemoryAddress (which the user expects), which causes performance issues with asType() > adaptations. 76: return MemoryHandles.filterCoordinates(handle, 0, ADDRESS_FILTER); The above comment needs updating to refer to `MemorySegmentProxy` and `MemorySegment`. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From aph at openjdk.java.net Fri Oct 9 17:38:16 2020 From: aph at openjdk.java.net (Andrew Haley) Date: Fri, 9 Oct 2020 17:38:16 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v5] In-Reply-To: References: Message-ID: <4NM17B6l4GvNgCbmmQTUcnfZTA6G-IEc85O8jH_q-xA=.63b10da7-bab7-44bc-a4c8-0a675aca45c0@github.com> On Fri, 9 Oct 2020 12:18:58 GMT, Andrew Haley wrote: >> Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains >> six commits: >> - Merge master >> - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check >> - Rebase >> - Merge master >> - Fix trailing whitespace issue >> - 8252204: AArch64: Implement SHA3 accelerator/intrinsic >> Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > Marked as reviewed by aph (Reviewer). I see Linux x64 failed. However, I don't seem to be able to withdraw my patch approval. However, please consider it withdrawn. ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From lancea at openjdk.java.net Fri Oct 9 19:15:14 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Fri, 9 Oct 2020 19:15:14 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v2] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Fri, 9 Oct 2020 10:28:34 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. The pull request contains one new commit since > the last revision: > 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size src/java.base/share/classes/java/util/jar/JarOutputStream.java line 84: > 82: * time. > 83: * > 84: * The compressed size field will be recalculated for compressed (i.e. I think the wording is better, but not sure we should state "compressed size field" as ZipEntry.setCompressedSize refers to "Sets the size of the compressed entry data." and "the compressed size to set" I think I would omit "field" from the above so it reads similar to "The compressed size will be..." or "The compressed size value will be..." ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From fyang at openjdk.java.net Sat Oct 10 02:56:17 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Sat, 10 Oct 2020 02:56:17 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v6] In-Reply-To: References: Message-ID: > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. > Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 > > Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. > For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. > "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. > > We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. > Patch passed jtreg tier1-3 tests with QEMU system emulator. > Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make > sure that there's no regression. > We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java > We measured the performance benefit with an aarch64 cycle-accurate simulator. > Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. > > For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on > real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is > detected. Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits: - Merge master - Merge master - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check - Rebase - Merge master - Fix trailing whitespace issue - 8252204: AArch64: Implement SHA3 accelerator/intrinsic Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com ------------- Changes: https://git.openjdk.java.net/jdk/pull/207/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=207&range=05 Stats: 1512 lines in 35 files changed: 1025 ins; 22 del; 465 mod Patch: https://git.openjdk.java.net/jdk/pull/207.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/207/head:pull/207 PR: https://git.openjdk.java.net/jdk/pull/207 From fyang at openjdk.java.net Sat Oct 10 06:16:17 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Sat, 10 Oct 2020 06:16:17 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v7] In-Reply-To: References: Message-ID: > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. > Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 > > Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. > For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. > "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. > > We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. > Patch passed jtreg tier1-3 tests with QEMU system emulator. > Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make > sure that there's no regression. > We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java > We measured the performance benefit with an aarch64 cycle-accurate simulator. > Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. > > For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on > real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is > detected. Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains eight commits: - Merge master - Merge master - Merge master - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check - Rebase - Merge master - Fix trailing whitespace issue - 8252204: AArch64: Implement SHA3 accelerator/intrinsic Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com ------------- Changes: https://git.openjdk.java.net/jdk/pull/207/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=207&range=06 Stats: 1512 lines in 35 files changed: 1025 ins; 22 del; 465 mod Patch: https://git.openjdk.java.net/jdk/pull/207.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/207/head:pull/207 PR: https://git.openjdk.java.net/jdk/pull/207 From fyang at openjdk.java.net Sat Oct 10 13:09:11 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Sat, 10 Oct 2020 13:09:11 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v5] In-Reply-To: <4NM17B6l4GvNgCbmmQTUcnfZTA6G-IEc85O8jH_q-xA=.63b10da7-bab7-44bc-a4c8-0a675aca45c0@github.com> References: <4NM17B6l4GvNgCbmmQTUcnfZTA6G-IEc85O8jH_q-xA=.63b10da7-bab7-44bc-a4c8-0a675aca45c0@github.com> Message-ID: <7CXYOoHPTvfS6YvwFjdlO27rQKRbDu3_QSGP7vDuyDs=.41789630-e05f-4f8a-8562-ad8bb74e12aa@github.com> On Fri, 9 Oct 2020 17:35:22 GMT, Andrew Haley wrote: > I see Linux x64 failed. However, I don't seem to be able to withdraw my patch approval. > However, please consider it withdrawn. Thanks for approving this patch. I checked the error messages and I think the failures were not caused by this patch. The failures has been fixed by the following two commits: commit ec41046c5ce7077eebf4a3c265f79c7fba33d916 8254348: Build fails when cds is disabled after JDK-8247536 commit aaa0a2a04792d7c84150e9d972790978ffcc6890 8254297: Zero and Minimal VMs are broken with undeclared identifier 'DerivedPointerTable' after JDK-8253180 The testing was triggered again automatically after I merge master and I see it passed now. Do you have any comments for the discussion here? https://github.com/openjdk/jdk/pull/207#issuecomment-701243662 Valerie Peng has checked the java security changes, i.e. src/java.base/share/classes/sun/security/provider/SHA3.java. Do you think we need another reviewer for this patch? ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From kcr at openjdk.java.net Sat Oct 10 13:19:09 2020 From: kcr at openjdk.java.net (Kevin Rushforth) Date: Sat, 10 Oct 2020 13:19:09 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v5] In-Reply-To: <4NM17B6l4GvNgCbmmQTUcnfZTA6G-IEc85O8jH_q-xA=.63b10da7-bab7-44bc-a4c8-0a675aca45c0@github.com> References: <4NM17B6l4GvNgCbmmQTUcnfZTA6G-IEc85O8jH_q-xA=.63b10da7-bab7-44bc-a4c8-0a675aca45c0@github.com> Message-ID: On Fri, 9 Oct 2020 17:35:22 GMT, Andrew Haley wrote: >> Marked as reviewed by aph (Reviewer). > > I see Linux x64 failed. However, I don't seem to be able to withdraw my patch approval. > However, please consider it withdrawn. @theRealAph if you still need to, you can withdraw your approval by reviewing it again and selecting "Request changes". ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From alanb at openjdk.java.net Sun Oct 11 15:44:13 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Sun, 11 Oct 2020 15:44:13 GMT Subject: RFR: 8253563: Change sun.security.jca.Providers.threadLists to be ThreadLocal In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 23:15:36 GMT, Valerie Peng wrote: > Could someone help reviewing this one-line change? This changes the provider list used during jar verification from > InheritableThreadLocal to ThreadLocal. Existing usage and handling uses this field as temporary thread-specific > provider list for jar verification. There seems to be no reason for it to be InheritableThreadLocal. Existing > regression tests pass with this change. > Thanks, > Valerie Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/570 From alanb at openjdk.java.net Sun Oct 11 17:26:10 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Sun, 11 Oct 2020 17:26:10 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v2] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Fri, 9 Oct 2020 10:28:34 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. src/java.base/share/classes/java/util/zip/ZipOutputStream.java line 189: > 187: * {@code ZipEntry.DEFLATED}) entries when {@link ZipEntry#setCompressedSize(long)} > 188: * has not been explicitly called on the {@code ZipEntry}. > 189: * Here's an alternative that might be a bit clearer to readers. "When writing a compressed (deflated) entry, and the compressed size has not been explicitly set with the setCompressedSize method, then the compressed size written to the entry's data descriptor will be its actual compressed size." I think we should put it after the "The default compression method .." sentence and move the sentence on the time stamp to the end. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Sun Oct 11 22:17:11 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Sun, 11 Oct 2020 22:17:11 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v2] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: <6RMxyurj4wb-oTldhToZhDebkRlbFrSTPzAWnhcni0Q=.457787d6-d1fe-455d-9ade-03e01c099db9@github.com> On Sun, 11 Oct 2020 17:22:58 GMT, Alan Bateman wrote: >> Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. > > src/java.base/share/classes/java/util/zip/ZipOutputStream.java line 189: > >> 187: * {@code ZipEntry.DEFLATED}) entries when {@link ZipEntry#setCompressedSize(long)} >> 188: * has not been explicitly called on the {@code ZipEntry}. >> 189: * > > Here's an alternative that might be a bit clearer to readers. > > "When writing a compressed (deflated) entry, and the compressed size has not been explicitly set with the > setCompressedSize method, then the compressed size written to the entry's data descriptor will be its actual compressed > size." I think we should put it after the "The default compression method .." sentence and move the sentence on the > time stamp to the end. I don't believe we discuss/reference the data descriptor for a Zip entry (outside of the PKWare Zip specification) so I am not sure we should reference it in the javadoc. Placing the sentence after "The default compression method will be used if no compression method was specified for the entry" makes sense ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From fyang at openjdk.java.net Mon Oct 12 07:05:16 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Mon, 12 Oct 2020 07:05:16 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v5] In-Reply-To: References: <4NM17B6l4GvNgCbmmQTUcnfZTA6G-IEc85O8jH_q-xA=.63b10da7-bab7-44bc-a4c8-0a675aca45c0@github.com> Message-ID: On Sat, 10 Oct 2020 13:15:51 GMT, Kevin Rushforth wrote: >> I see Linux x64 failed. However, I don't seem to be able to withdraw my patch approval. >> However, please consider it withdrawn. > > @theRealAph if you still need to, you can withdraw your approval by reviewing it again and selecting "Request changes". > I have looked at the java security changes, i.e. src/java.base/share/classes/sun/security/provider/SHA3.java. It looks > fine. @valeriepeng : I see you are not listed under "Reviewers" commit message part, could you please press the magic button(s)(approve?) so you get the credit? Thanks. ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From mcimadamore at openjdk.java.net Mon Oct 12 10:50:48 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Mon, 12 Oct 2020 10:50:48 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v6] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Tweak referenced to MemoryAddressProxy in Utils.java ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/9b3fc227..770b1e9c Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=04-05 Stats: 6 lines in 1 file changed: 0 ins; 1 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From simonis at openjdk.java.net Mon Oct 12 11:44:28 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Mon, 12 Oct 2020 11:44:28 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/520/files - new: https://git.openjdk.java.net/jdk/pull/520/files/2a9427ef..988f3ef9 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=01-02 Stats: 12 lines in 2 files changed: 0 ins; 0 del; 12 mod Patch: https://git.openjdk.java.net/jdk/pull/520.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/520/head:pull/520 PR: https://git.openjdk.java.net/jdk/pull/520 From ihse at openjdk.java.net Mon Oct 12 11:45:15 2020 From: ihse at openjdk.java.net (Magnus Ihse Bursie) Date: Mon, 12 Oct 2020 11:45:15 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v6] In-Reply-To: References: Message-ID: On Mon, 12 Oct 2020 10:50:48 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation >> (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from >> multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee >> that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class >> has been added, which defines several useful dereference routines; these are really just thin wrappers around memory >> access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not >> the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link >> to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit >> of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which >> wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as >> dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability >> in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; >> secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can >> use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done >> by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided >> below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be >> happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, >> Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd >> like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio >> Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative >> to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a >> carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access >> base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte >> offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. >> `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which >> it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both >> `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients >> can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory >> access var handle support. >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to >> achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment >> is shared, it would be possible for a thread to close it while another is accessing it. After considering several >> options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he >> reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world >> (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a >> close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and >> the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). >> It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of >> these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, >> we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to >> whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of >> stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, >> once we detect that a thread is accessing the very segment we're about to close, what should happen? We first >> experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it >> fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the >> machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to >> minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread >> is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and >> try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should >> be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single >> place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in >> addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` >> annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) >> class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, >> like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is >> tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during >> access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory >> access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead >> of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a >> `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed >> successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two >> implementations, one for confined segments and one for shared segments; the main difference between the two is what >> happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared >> segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or >> `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` >> state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### >> Memory access var handles overhaul The key realization here was that if all memory access var handles took a >> coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle >> form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var >> handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that >> e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the >> implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level >> access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see >> here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, >> since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` >> functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the >> microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared >> segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - >> https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Tweak referenced to MemoryAddressProxy in Utils.java Build changes look good. ------------- Marked as reviewed by ihse (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/548 From simonis at openjdk.java.net Mon Oct 12 11:49:12 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Mon, 12 Oct 2020 11:49:12 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v2] In-Reply-To: <6RMxyurj4wb-oTldhToZhDebkRlbFrSTPzAWnhcni0Q=.457787d6-d1fe-455d-9ade-03e01c099db9@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <6RMxyurj4wb-oTldhToZhDebkRlbFrSTPzAWnhcni0Q=.457787d6-d1fe-455d-9ade-03e01c099db9@github.com> Message-ID: On Sun, 11 Oct 2020 22:14:18 GMT, Lance Andersen wrote: >> src/java.base/share/classes/java/util/zip/ZipOutputStream.java line 189: >> >>> 187: * {@code ZipEntry.DEFLATED}) entries when {@link ZipEntry#setCompressedSize(long)} >>> 188: * has not been explicitly called on the {@code ZipEntry}. >>> 189: * >> >> Here's an alternative that might be a bit clearer to readers. >> >> "When writing a compressed (deflated) entry, and the compressed size has not been explicitly set with the >> setCompressedSize method, then the compressed size written to the entry's data descriptor will be its actual compressed >> size." I think we should put it after the "The default compression method .." sentence and move the sentence on the >> time stamp to the end. > > I don't believe we discuss/reference the data descriptor for a Zip entry (outside of the PKWare Zip specification) so I > am not sure we should reference it in the javadoc. > Placing the sentence after "The default compression method will be used if no compression method was specified for the > entry" makes sense Thanks for your input. I've tried to somehow merge both suggestions :) Did you had a chance to look at the CSR? I think somebody has to review it before I can move it to "Finalized". Thank you and best regards, Volker ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From alanb at openjdk.java.net Mon Oct 12 12:51:14 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Mon, 12 Oct 2020 12:51:14 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Mon, 12 Oct 2020 11:44:28 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. The pull request contains one new commit since > the last revision: > 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size src/java.base/share/classes/java/util/jar/JarOutputStream.java line 86: > 84: * compressed size will be set to the actual compressed size after > 85: * deflation. The current time will be used if the entry has no set > 86: * modification time. Thanks for combining the wording, I think this looks good. I'd probably drop "i.e." so that it's just "DEFLATED" parentheses. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From alanb at openjdk.java.net Mon Oct 12 12:57:29 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Mon, 12 Oct 2020 12:57:29 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Mon, 12 Oct 2020 11:44:28 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. The pull request contains one new commit since > the last revision: > 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size src/java.base/share/classes/java/util/zip/ZipEntry.java line 56: > 54: long size = -1; // uncompressed size of entry data > 55: long csize = -1; // compressed size of entry data > 56: boolean manual_csize = false; // Only true if csize was explicitely set by a call to setCompressedSize() Minor nit but this should probably be "csizeSet" as it's not too common to have underscore in field names. src/java.base/share/classes/java/util/zip/ZipOutputStream.java line 216: > 214: e.flag = 8; > 215: } > 216: else if (!e.manual_csize) { I assume the existing expression that set if csize has been set so that we don't set the flag to 8 in two branches. test/jdk/java/util/zip/CopyZipFile.java line 55: > 53: File f = new File(zipFile); > 54: f.deleteOnExit(); > 55: OutputStream os = new FileOutputStream(f); Can you use try-with-resources here so that a test failure will close the file. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From Alan.Bateman at oracle.com Mon Oct 12 13:41:13 2020 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 12 Oct 2020 14:41:13 +0100 Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v2] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <6RMxyurj4wb-oTldhToZhDebkRlbFrSTPzAWnhcni0Q=.457787d6-d1fe-455d-9ade-03e01c099db9@github.com> Message-ID: <73ae6ebc-a9fd-16d3-32a8-88877b096bd2@oracle.com> On 12/10/2020 12:49, Volker Simonis wrote: > : > Did you had a chance to look at the CSR? I think somebody has to review it before I can move it to "Finalized". > The updated javadoc looks good. Once Lance has seen it then I think the CSR would be updated so that it's in sync with the proposal and then we add ourselves as Reviewer and finalize. -Alan From dfuchs at openjdk.java.net Mon Oct 12 13:50:30 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Mon, 12 Oct 2020 13:50:30 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v3] In-Reply-To: References: Message-ID: > Hi, > > This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of > synchronized monitors. > Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are > outlined below: > 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: > `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing > ( a snapshot of the data is taken before printing instead) > > 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: > `MeteredStream::close` was missing synchronization: it is now protected by the lock > > 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: > `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. > Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that > `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` > is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user > code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and > reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe > there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` > method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of > casting to PrintStream. > > 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: > Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. > There is no need for a synchronized here as the method is only called from a method that already > holds the lock. > > 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: > Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. > This method is only called from methods already protected by the lock. > Also the method has been made private. > > 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java > `queuedForCleanup` is made volatile as it is read and written directly from outside the class > and without protection by the `KeepAliveCleanerEntry`. > Lock protection is also added to `close()`, which was missing it. > Some methods that have no lock protection and did not need it because only called from > within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` > > 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for > `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have > acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called > from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` > > 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java > Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` > to help track the fact that AuthenticationInfo::setHeaders is only called while > holding the lock. > Synchronization was also removed from some method that didn't need it because only > called from within code blocks protected by the lock: > `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). > > 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. Daniel Fuchs has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: - 8229867: Re-examine synchronization usages in http and https protocol handlers Incorporated review feedback - Merge - 8229867: Re-examine synchronization usages in http and https protocol handlers Incorporated review feedback - 8229867: Re-examine synchronization usages in http and https protocol handlers ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/558/files - new: https://git.openjdk.java.net/jdk/pull/558/files/ddfa2e6c..d8fa0439 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=558&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=558&range=01-02 Stats: 23530 lines in 473 files changed: 14274 ins; 6170 del; 3086 mod Patch: https://git.openjdk.java.net/jdk/pull/558.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/558/head:pull/558 PR: https://git.openjdk.java.net/jdk/pull/558 From dfuchs at openjdk.java.net Mon Oct 12 13:50:31 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Mon, 12 Oct 2020 13:50:31 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v2] In-Reply-To: References: Message-ID: On Fri, 9 Oct 2020 13:22:08 GMT, Alan Bateman wrote: >> Daniel Fuchs has updated the pull request incrementally with one additional commit since the last revision: >> >> 8229867: Re-examine synchronization usages in http and https protocol handlers >> >> Incorporated review feedback > > src/java.base/share/classes/sun/net/www/http/HttpCapture.java line 59: > >> 57: // Although accessing files could result in blocking operations, >> 58: // HttpCapture is a corner case; there seem no urgent need to convert >> 59: // this class to using java.util.concurrent.locks at this time. > > The updated patch looks good but I think this comment needs another iteration to ensure that it doesn't confuse future > maintainers. You could drop it or else replace it with something simple that says that HttpCapture does blocking I/O > operations while holding monitors but it's not a concern because it rarely used. Updated as requested. ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From alanb at openjdk.java.net Mon Oct 12 14:16:13 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Mon, 12 Oct 2020 14:16:13 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v3] In-Reply-To: References: Message-ID: <3ZfHI2iWTIkNhZHDEAUTXxBDfajuDu21McbJWNmf-5c=.007587e5-7d25-432d-8d77-a006524833cd@github.com> On Mon, 12 Oct 2020 13:50:30 GMT, Daniel Fuchs wrote: >> Hi, >> >> This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of >> synchronized monitors. >> Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are >> outlined below: >> 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: >> `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing >> ( a snapshot of the data is taken before printing instead) >> >> 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: >> `MeteredStream::close` was missing synchronization: it is now protected by the lock >> >> 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: >> `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. >> Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that >> `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` >> is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user >> code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and >> reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe >> there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` >> method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of >> casting to PrintStream. >> >> 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: >> Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. >> There is no need for a synchronized here as the method is only called from a method that already >> holds the lock. >> >> 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: >> Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. >> This method is only called from methods already protected by the lock. >> Also the method has been made private. >> >> 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java >> `queuedForCleanup` is made volatile as it is read and written directly from outside the class >> and without protection by the `KeepAliveCleanerEntry`. >> Lock protection is also added to `close()`, which was missing it. >> Some methods that have no lock protection and did not need it because only called from >> within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` >> >> 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for >> `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have >> acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called >> from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` >> >> 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java >> Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` >> to help track the fact that AuthenticationInfo::setHeaders is only called while >> holding the lock. >> Synchronization was also removed from some method that didn't need it because only >> called from within code blocks protected by the lock: >> `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). >> >> 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > Daniel Fuchs has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev > excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since > the last revision: > - 8229867: Re-examine synchronization usages in http and https protocol handlers > > Incorporated review feedback > - Merge > - 8229867: Re-examine synchronization usages in http and https protocol handlers > > Incorporated review feedback > - 8229867: Re-examine synchronization usages in http and https protocol handlers Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From fyang at openjdk.java.net Mon Oct 12 14:47:32 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Mon, 12 Oct 2020 14:47:32 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v8] In-Reply-To: References: Message-ID: <76qazRT5aX06rurPVGQmtfH2af9_l7DEdy_mGyF7BQQ=.4dd45e3f-b7a4-44c9-9824-4daab9b7dc3b@github.com> > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. > Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 > > Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. > For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. > "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. > > We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. > Patch passed jtreg tier1-3 tests with QEMU system emulator. > Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make > sure that there's no regression. > We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java > We measured the performance benefit with an aarch64 cycle-accurate simulator. > Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. > > For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on > real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is > detected. Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 10 commits: - Remove unnecessary code changes in vm_version_aarch64.cpp - Merge master - Merge master - Merge master - Merge master - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check - Rebase - Merge master - Fix trailing whitespace issue - 8252204: AArch64: Implement SHA3 accelerator/intrinsic Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com ------------- Changes: https://git.openjdk.java.net/jdk/pull/207/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=207&range=07 Stats: 1498 lines in 35 files changed: 1011 ins; 22 del; 465 mod Patch: https://git.openjdk.java.net/jdk/pull/207.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/207/head:pull/207 PR: https://git.openjdk.java.net/jdk/pull/207 From lancea at openjdk.java.net Mon Oct 12 15:40:19 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 12 Oct 2020 15:40:19 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Mon, 12 Oct 2020 11:44:28 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. The pull request contains one new commit since > the last revision: > 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size src/java.base/share/classes/java/util/jar/JarOutputStream.java line 81: > 79: * any previous entry. The default compression method will be > 80: * used if no compression method was specified for the entry. > 81: * When writing a compressed (i.e. {@code ZipEntry.DEFLATED})) I would leave as DEFLATED given it is used this way in other ZipOutputStream methods and I would also remove the "i.e." ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Mon Oct 12 15:40:20 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 12 Oct 2020 15:40:20 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Mon, 12 Oct 2020 12:48:36 GMT, Alan Bateman wrote: >> Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. The pull request contains one new commit since >> the last revision: >> 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size > > src/java.base/share/classes/java/util/jar/JarOutputStream.java line 86: > >> 84: * compressed size will be set to the actual compressed size after >> 85: * deflation. The current time will be used if the entry has no set >> 86: * modification time. > > Thanks for combining the wording, I think this looks good. I'd probably drop "i.e." so that it's just "DEFLATED" > parentheses. I might consider making "The current time..." its own paragraph separate from the compression discussion as I think it would be clearer that way. > src/java.base/share/classes/java/util/zip/ZipOutputStream.java line 216: > >> 214: e.flag = 8; >> 215: } >> 216: else if (!e.manual_csize) { > > I assume the existing expression that set if csize has been set so that we don't set the flag to 8 in two branches. I was thinking about that myself ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Mon Oct 12 15:45:24 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 12 Oct 2020 15:45:24 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Mon, 12 Oct 2020 11:44:28 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. The pull request contains one new commit since > the last revision: > 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size I think we are getting close. Once the javadoc is cleaned up along with the minor changes recommended we should be in good shape and can then review the CSR ------------- Changes requested by lancea (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/520 From alanb at openjdk.java.net Mon Oct 12 15:52:13 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Mon, 12 Oct 2020 15:52:13 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Mon, 12 Oct 2020 15:35:12 GMT, Lance Andersen wrote: >> Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. The pull request contains one new commit since >> the last revision: >> 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size > > src/java.base/share/classes/java/util/jar/JarOutputStream.java line 81: > >> 79: * any previous entry. The default compression method will be >> 80: * used if no compression method was specified for the entry. >> 81: * When writing a compressed (i.e. {@code ZipEntry.DEFLATED})) > > I would leave as DEFLATED given it is used this way in other ZipOutputStream methods and I would also remove the "i.e." I would prefer to keep it as "When writing as compressed (DEFLATED) ..." to keep the terminology consistent when linking to the setCompressedMethod. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Mon Oct 12 15:57:15 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 12 Oct 2020 15:57:15 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: <6HxY6XtWrf29dTXOd1ptm23hMncg7AIhqLvX0xrMO4A=.a649a6a8-19ea-4c2f-968a-711a7dbfa4cd@github.com> On Mon, 12 Oct 2020 15:49:29 GMT, Alan Bateman wrote: >> src/java.base/share/classes/java/util/jar/JarOutputStream.java line 81: >> >>> 79: * any previous entry. The default compression method will be >>> 80: * used if no compression method was specified for the entry. >>> 81: * When writing a compressed (i.e. {@code ZipEntry.DEFLATED})) >> >> I would leave as DEFLATED given it is used this way in other ZipOutputStream methods and I would also remove the "i.e." > > I would prefer to keep it as "When writing as compressed (DEFLATED) ..." to keep the terminology consistent when > linking to the setCompressedMethod. Agree, no need for ZipEntry.DEFLATED which is what I was trying to say (sorry if that was not clear). Only suggesting ZipEntry.DEFLATED -> DEFLATED and remove the ".i.e." ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From valeriep at openjdk.java.net Mon Oct 12 17:00:13 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Mon, 12 Oct 2020 17:00:13 GMT Subject: Integrated: 8253563: Change sun.security.jca.Providers.threadLists to be ThreadLocal In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 23:15:36 GMT, Valerie Peng wrote: > Could someone help reviewing this one-line change? This changes the provider list used during jar verification from > InheritableThreadLocal to ThreadLocal. Existing usage and handling uses this field as temporary thread-specific > provider list for jar verification. There seems to be no reason for it to be InheritableThreadLocal. Existing > regression tests pass with this change. > Thanks, > Valerie This pull request has now been integrated. Changeset: df1f132b Author: Valerie Peng URL: https://git.openjdk.java.net/jdk/commit/df1f132b Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod 8253563: Change sun.security.jca.Providers.threadLists to be ThreadLocal Reviewed-by: alanb ------------- PR: https://git.openjdk.java.net/jdk/pull/570 From valeriep at openjdk.java.net Mon Oct 12 17:38:14 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Mon, 12 Oct 2020 17:38:14 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 06:51:08 GMT, Anthony Scarpino wrote: >> 8253821: Improve ByteBuffer performance with GCM > > Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: > > Xuelei comments src/java.base/share/classes/com/sun/crypto/provider/GCTR.java line 146: > 144: private static final int MAX_LEN = 1024; > 145: > 146: int doFinal(ByteBuffer src, ByteBuffer dst) throws IllegalBlockSizeException { Conventionally, we put doFinal() method after update() method. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From mcimadamore at openjdk.java.net Mon Oct 12 17:58:54 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Mon, 12 Oct 2020 17:58:54 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v7] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Tweak support for mapped memory segments ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/770b1e9c..75e406c0 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=05-06 Stats: 543 lines in 13 files changed: 336 ins; 151 del; 56 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Mon Oct 12 18:10:18 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Mon, 12 Oct 2020 18:10:18 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v6] In-Reply-To: References: Message-ID: On Mon, 12 Oct 2020 11:42:04 GMT, Magnus Ihse Bursie wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Tweak referenced to MemoryAddressProxy in Utils.java > > Build changes look good. I've just uploaded a biggie update to the foreign memory access support. While doing performance evaluation, we have realized that mixing a multi-level hierarchy (`MappedMemorySegment extends MemorySegments`) with exact invoke semantics of `VarHandle` and `MethodHandle` is not a good match and can lead to great performance degradation for seemingly "correct" code. While some of this can be attributed to the `VarHandle` API, or to the fact that the so called "generic" invocation path should not be that slow in case where the parameters are clearly related, it seems smelly that a primitive API such as `MemorySegment` should give raise to such issues. We have therefore decided to drop the `MappedMemorySegment` - this means that there's only one memory segment type users can deal with: `MemorySegment` - and no chance for mistakes. Of course `MappedMemorySegment` has been primarily introduces to allow for operations which were previously possible on `MappedByteBuffer` such as `force`. To support these use cases, a separate class has been introduced, namely `MappedMemorySegments` (note the trailing `S`). This class contains a bunch of static methods which can be used to achieve the desired effects, without polluting the `MemorySegment` API. A new method has been added on `MemorySegment` which returns an optional file descriptor; this might be useful for clients which want to guess whether a segment is in fact a mapped segment, or if they need (e.g. in Windows) the file descriptor to do some other kind of low level op. I think this approach is more true to the goals and spirit of the Foreign Memory Access API, and it also offers some ways to improve over the existing API: for instance, the only reason why the `MemorySegment::spliterator` method was a static method was that we needed inference, so that we could return either a `Spliterator` or a `Spliterator`. All of that is gone now, so the method can return to be what it morally always has been: an instance method on `MemorySegment`. Updated javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v2/javadoc/jdk/incubator/foreign/package-summary.html Updated specdiff: http://cr.openjdk.java.net/~mcimadamore/8254162_v2/specdiff/overview-summary.html ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From valeriep at openjdk.java.net Mon Oct 12 19:24:16 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Mon, 12 Oct 2020 19:24:16 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: <2udH_wWPCcqka4WmVmcLlezitoprq0c_w9YHxAbmP1E=.975d839f-ce4f-4056-9487-a3233e758331@github.com> On Thu, 8 Oct 2020 06:51:08 GMT, Anthony Scarpino wrote: >> 8253821: Improve ByteBuffer performance with GCM > > Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: > > Xuelei comments src/java.base/share/classes/com/sun/crypto/provider/GCTR.java line 165: > 163: } > 164: > 165: int update(ByteBuffer src, ByteBuffer dst) { Based on the impl in GaloisCounterMode, this is only called when src.remaining() >= 128. Perhaps documenting the conditions here as there are no checks on the src/dst sizes as in the byte[] case. src/java.base/share/classes/com/sun/crypto/provider/GCTR.java line 237: > 235: encrypt(in, offset, processed, out, 0); > 236: dst.get(out, 0, processed); > 237: return len; This block of code looks strange? len = inLen % AES_BLOCK_SIZE => len must be between 0...AES_BLOCK_SIZE-1, I think you meant to use len = (inLen - inLen% AES_BLOCK_SIZE) dst.get(...) should be dst.put(...) ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From mcimadamore at openjdk.java.net Mon Oct 12 20:18:50 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Mon, 12 Oct 2020 20:18:50 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v8] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Simplify example in the toplevel javadoc ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/75e406c0..d14d06a4 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=07 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=06-07 Stats: 14 lines in 1 file changed: 4 ins; 5 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From valeriep at openjdk.java.net Mon Oct 12 21:01:15 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Mon, 12 Oct 2020 21:01:15 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 01:22:09 GMT, Anthony Scarpino wrote: >> src/java.base/share/classes/com/sun/crypto/provider/AESCipher.java line 744: >> >>> 742: } else { >>> 743: return core.doFinal(input, output); >>> 744: } >> >> It seems this block is the only difference between this method and CipherSpi.bufferCrypt(). Have you considered moving >> this special handling to the overridden engineDoFinal(...) method and not duplicating the whole CipherSpi.bufferCrypt() >> method here? BTW, instead of using the generic update/doFinal name and then commenting them for GCM usage only, perhaps >> it's more enticing to name them as gcmUpdate/gcmDoFinal? > > I didn't see a way to override this because CipherSpi is a public class, any methods I added would become a new API. > Also bufferCrypt is private, so I had to copy it. CipherSpi does not know which mode is being used, but AESCipher > does. Maybe I'm missing something, I'd rather it not be a copy, but I couldn't see a better way. If you have a > specific idea, please give me details. How about overriding protected int engineDoFinal(ByteBuffer input, ByteBuffer output) throws ... { if (core.getMode() == CipherCore.GCM_MODE && !input.hasArray()) { // call your own optimized byte buffer code path } else { super.engineDoFinal(input, output); } } Would this work? If yes, then no need to duplicate the whole bufferCrypt() method. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From valeriep at openjdk.java.net Mon Oct 12 22:05:32 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Mon, 12 Oct 2020 22:05:32 GMT Subject: RFR: 8199697: FIPS 186-4 RSA Key Generation [v2] In-Reply-To: References: Message-ID: > Could someone please help review this RFE? Update existing RSA key pair generation code following the guidelines from > FIPS 186-4 and FIPS 186-5 (draft). Current proposed changes updates the prime generation code (for P, Q) based on FIPS > 186-4 B.3.3 when keysize and public exponent met the requirements set in FIPS 186-4/5. Thanks, > Valerie Valerie Peng has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since the last revision: - Merge branch 'master' into JDK-8199697 - 8199697: FIPS 186-4 RSA Key Generation Changed RSA key pair generation code following the guidelines from FIPS 186-4. ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/420/files - new: https://git.openjdk.java.net/jdk/pull/420/files/441c114a..358fee06 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=420&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=420&range=00-01 Stats: 44669 lines in 2466 files changed: 22664 ins; 10552 del; 11453 mod Patch: https://git.openjdk.java.net/jdk/pull/420.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/420/head:pull/420 PR: https://git.openjdk.java.net/jdk/pull/420 From valeriep at openjdk.java.net Mon Oct 12 23:00:12 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Mon, 12 Oct 2020 23:00:12 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v5] In-Reply-To: References: <4NM17B6l4GvNgCbmmQTUcnfZTA6G-IEc85O8jH_q-xA=.63b10da7-bab7-44bc-a4c8-0a675aca45c0@github.com> Message-ID: On Mon, 12 Oct 2020 07:02:05 GMT, Fei Yang wrote: > > > > I have looked at the java security changes, i.e. src/java.base/share/classes/sun/security/provider/SHA3.java. It looks > > fine. > > @valeriepeng : I see you are not listed under "Reviewers" commit message part, could you please press the magic > button(s)(approve?) so you get the credit? Thanks. It's fine, the part I reviewed is only a small part of the changes, so I will leave the reviewer approval upto the hotspot team. Thanks, Valerie ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From mcimadamore at openjdk.java.net Tue Oct 13 10:27:43 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 13 Oct 2020 10:27:43 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v9] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 10 additional commits since the last revision: - Merge branch 'master' into 8254162 - Simplify example in the toplevel javadoc - Tweak support for mapped memory segments - Tweak referenced to MemoryAddressProxy in Utils.java - Fix performance issue with "small" segment mismatch - Address review comments - Fix indent in GensrcScopedMemoryAccess.gmk - Address review comments - Add modified files - RFR 8254162: Implementation of Foreign-Memory Access API (Third Incubator) This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible. Thanks Maurizio Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html Specdiff: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html CSR: https://bugs.openjdk.java.net/browse/JDK-8254163 * `MemorySegment` * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) * added a no-arg factory for a native restricted segment representing entire native heap * rename `withOwnerThread` to `handoff` * add new `share` method, to create shared segments * add new `registerCleaner` method, to register a segment against a cleaner * add more helpers to create arrays from a segment e.g. `toIntArray` * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) * `MemoryAddress` * drop `segment` accessor * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment * `MemoryAccess` * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). * `MemoryHandles` * drop `withOffset` combinator * drop `withStride` combinator * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. * `Addressable` * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. * `MemoryLayouts` * A new layout, for machine addresses, has been added to the mix. There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/d14d06a4..8815d941 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=08 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=07-08 Stats: 22231 lines in 447 files changed: 12727 ins; 6345 del; 3159 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Tue Oct 13 11:06:32 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 13 Oct 2020 11:06:32 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v10] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Remove spurious check on MemoryScope::confineTo Added tests to make sure no spurious exception is thrown when: * handing off a segment from A to A * sharing an already shared segment ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/8815d941..8fb8ff2f Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=09 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=08-09 Stats: 27 lines in 2 files changed: 16 ins; 8 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Tue Oct 13 11:23:33 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 13 Oct 2020 11:23:33 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v11] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 12 additional commits since the last revision: - Merge branch 'master' into 8254162 - Remove spurious check on MemoryScope::confineTo Added tests to make sure no spurious exception is thrown when: * handing off a segment from A to A * sharing an already shared segment - Merge branch 'master' into 8254162 - Simplify example in the toplevel javadoc - Tweak support for mapped memory segments - Tweak referenced to MemoryAddressProxy in Utils.java - Fix performance issue with "small" segment mismatch - Address review comments - Fix indent in GensrcScopedMemoryAccess.gmk - Address review comments - ... and 2 more: https://git.openjdk.java.net/jdk/compare/f7d78c01...e866bb23 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/8fb8ff2f..e866bb23 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=10 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=09-10 Stats: 605 lines in 49 files changed: 426 ins; 116 del; 63 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From simonis at openjdk.java.net Tue Oct 13 11:40:36 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Tue, 13 Oct 2020 11:40:36 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v4] In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) Volker Simonis has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains one additional commit since the last revision: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/520/files - new: https://git.openjdk.java.net/jdk/pull/520/files/988f3ef9..3edcf72e Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=02-03 Stats: 43936 lines in 2282 files changed: 22490 ins; 10444 del; 11002 mod Patch: https://git.openjdk.java.net/jdk/pull/520.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/520/head:pull/520 PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Tue Oct 13 11:40:36 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Tue, 13 Oct 2020 11:40:36 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v3] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Mon, 12 Oct 2020 15:42:45 GMT, Lance Andersen wrote: >> Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. The pull request contains one new commit since >> the last revision: >> 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size > > I think we are getting close. Once the javadoc is cleaned up along with the minor changes recommended we should be > in good shape and can then review the CSR I hope I've addressed all your comments and suggestion with the latest PR. I've also updated the CSR to reflect the latest version of the API changes from the PR. Please have a look. Thank you and best regards, Volker ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From weijun at openjdk.java.net Tue Oct 13 12:35:26 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 13 Oct 2020 12:35:26 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v5] In-Reply-To: References: Message-ID: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: support rfc5652 CMSAlgorithmProtection ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/322/files - new: https://git.openjdk.java.net/jdk/pull/322/files/fecf30d7..81513907 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=03-04 Stats: 72 lines in 5 files changed: 65 ins; 0 del; 7 mod Patch: https://git.openjdk.java.net/jdk/pull/322.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/322/head:pull/322 PR: https://git.openjdk.java.net/jdk/pull/322 From chegar at openjdk.java.net Tue Oct 13 13:12:14 2020 From: chegar at openjdk.java.net (Chris Hegarty) Date: Tue, 13 Oct 2020 13:12:14 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v3] In-Reply-To: References: Message-ID: On Mon, 12 Oct 2020 13:50:30 GMT, Daniel Fuchs wrote: >> Hi, >> >> This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of >> synchronized monitors. >> Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are >> outlined below: >> 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: >> `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing >> ( a snapshot of the data is taken before printing instead) >> >> 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: >> `MeteredStream::close` was missing synchronization: it is now protected by the lock >> >> 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: >> `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. >> Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that >> `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` >> is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user >> code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and >> reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe >> there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` >> method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of >> casting to PrintStream. >> >> 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: >> Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. >> There is no need for a synchronized here as the method is only called from a method that already >> holds the lock. >> >> 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: >> Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. >> This method is only called from methods already protected by the lock. >> Also the method has been made private. >> >> 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java >> `queuedForCleanup` is made volatile as it is read and written directly from outside the class >> and without protection by the `KeepAliveCleanerEntry`. >> Lock protection is also added to `close()`, which was missing it. >> Some methods that have no lock protection and did not need it because only called from >> within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` >> >> 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for >> `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have >> acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called >> from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` >> >> 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java >> Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` >> to help track the fact that AuthenticationInfo::setHeaders is only called while >> holding the lock. >> Synchronization was also removed from some method that didn't need it because only >> called from within code blocks protected by the lock: >> `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). >> >> 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > Daniel Fuchs has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev > excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since > the last revision: > - 8229867: Re-examine synchronization usages in http and https protocol handlers > > Incorporated review feedback > - Merge > - 8229867: Re-examine synchronization usages in http and https protocol handlers > > Incorporated review feedback > - 8229867: Re-examine synchronization usages in http and https protocol handlers LGTM. ------------- Marked as reviewed by chegar (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/558 From michaelm at openjdk.java.net Tue Oct 13 13:28:17 2020 From: michaelm at openjdk.java.net (Michael McMahon) Date: Tue, 13 Oct 2020 13:28:17 GMT Subject: RFR: 8229867: Re-examine synchronization usages in http and https protocol handlers [v3] In-Reply-To: References: Message-ID: On Mon, 12 Oct 2020 13:50:30 GMT, Daniel Fuchs wrote: >> Hi, >> >> This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of >> synchronized monitors. >> Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are >> outlined below: >> 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: >> `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing >> ( a snapshot of the data is taken before printing instead) >> >> 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: >> `MeteredStream::close` was missing synchronization: it is now protected by the lock >> >> 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: >> `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. >> Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that >> `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` >> is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user >> code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and >> reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe >> there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` >> method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of >> casting to PrintStream. >> >> 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: >> Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. >> There is no need for a synchronized here as the method is only called from a method that already >> holds the lock. >> >> 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: >> Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. >> This method is only called from methods already protected by the lock. >> Also the method has been made private. >> >> 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java >> `queuedForCleanup` is made volatile as it is read and written directly from outside the class >> and without protection by the `KeepAliveCleanerEntry`. >> Lock protection is also added to `close()`, which was missing it. >> Some methods that have no lock protection and did not need it because only called from >> within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` >> >> 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for >> `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have >> acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called >> from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` >> >> 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java >> Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` >> to help track the fact that AuthenticationInfo::setHeaders is only called while >> holding the lock. >> Synchronization was also removed from some method that didn't need it because only >> called from within code blocks protected by the lock: >> `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` >> `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). >> >> 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. >> >> 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: >> synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > Daniel Fuchs has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev > excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since > the last revision: > - 8229867: Re-examine synchronization usages in http and https protocol handlers > > Incorporated review feedback > - Merge > - 8229867: Re-examine synchronization usages in http and https protocol handlers > > Incorporated review feedback > - 8229867: Re-examine synchronization usages in http and https protocol handlers Marked as reviewed by michaelm (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From mcimadamore at openjdk.java.net Tue Oct 13 13:29:28 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 13 Oct 2020 13:29:28 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) Message-ID: This patch contains the changes associated with the first incubation round of the foreign linker access API incubation (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio Webrev: http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html Specdiff (relative to [3]): http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html CSR: https://bugs.openjdk.java.net/browse/JDK-8254232 ### API Changes The API changes are actually rather slim: * `LibraryLookup` * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. * `FunctionDescriptor` * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. * `CLinker` * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. * `NativeScope` * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. * `MemorySegment` * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. ### Safety The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more readings on the internals of the foreign linker support, please refer to [5]. #### Test changes Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - https://git.openjdk.java.net/jdk/pull/548 [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html ------------- Commit messages: - Fix more whitespaces - Fix whitespaces - Remove rejected file - More updates - Add new files - Merge with master - Merge branch 'master' into 8254162 - Remove spurious check on MemoryScope::confineTo - Merge branch 'master' into 8254162 - Simplify example in the toplevel javadoc - ... and 8 more: https://git.openjdk.java.net/jdk/compare/5d6a6255...7cf0ef09 Changes: https://git.openjdk.java.net/jdk/pull/634/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8254231 Stats: 75468 lines in 264 files changed: 72636 ins; 1596 del; 1236 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From weijun at openjdk.java.net Tue Oct 13 13:34:27 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 13 Oct 2020 13:34:27 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Weijun Wang has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: support rfc6211 CMSAlgorithmProtection ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/322/files - new: https://git.openjdk.java.net/jdk/pull/322/files/81513907..ffaae532 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=04-05 Stats: 0 lines in 0 files changed: 0 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/322.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/322/head:pull/322 PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Tue Oct 13 13:34:27 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 13 Oct 2020 13:34:27 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: On Sun, 4 Oct 2020 14:09:26 GMT, Weijun Wang wrote: >> Changes requested by alanb (Reviewer). > > Note: I force pushed a new commit to correct a typo in the summary line. Add support for [RFC 6211: Cryptographic Message Syntax (CMS) Algorithm Identifier Protection Attribute](https://tools.ietf.org/html/rfc6211) to protect against algorithm substitution attacks. This attribute is signed and it contains copies of digestAlgorithm and signatureAlgorithm which are unprotected in SignerInfo. Before this enhancement, the two algorithms can be implied from the signature itself (i.e. if you change any of them the signature size would not match or the key will not decrypt). However, with the introduction of RSASSA-PSS, the signature algorithm can be modified and it still looks like the signature is valid. This particular case is [described in the RFC](https://tools.ietf.org/html/rfc6211#page-5): signatureAlgorithm has been protected by implication in the past. The use of an RSA public key implied that the RSA v1.5 signature algorithm was being used. The hash algorithm and this fact could be checked by the internal padding defined. This is no longer true with the addition of the RSA-PSS signature algorithms. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Tue Oct 13 13:34:27 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 13 Oct 2020 13:34:27 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: On Tue, 13 Oct 2020 13:24:50 GMT, Weijun Wang wrote: >> Note: I force pushed a new commit to correct a typo in the summary line. > > Add support for [RFC 6211: Cryptographic Message Syntax (CMS) Algorithm Identifier Protection > Attribute](https://tools.ietf.org/html/rfc6211) to protect against algorithm substitution attacks. This attribute is > signed and it contains copies of digestAlgorithm and signatureAlgorithm which are unprotected in SignerInfo. Before > this enhancement, the two algorithms can be implied from the signature itself (i.e. if you change any of them the > signature size would not match or the key will not decrypt). However, with the introduction of RSASSA-PSS, the > signature algorithm can be modified and it still looks like the signature is valid. This particular case is [described > in the RFC](https://tools.ietf.org/html/rfc6211#page-5): > signatureAlgorithm has been protected by implication in the past. > The use of an RSA public key implied that the RSA v1.5 signature > algorithm was being used. The hash algorithm and this fact could > be checked by the internal padding defined. This is no longer > true with the addition of the RSA-PSS signature algorithms. A force push to fix the RFC number typo in the latest commit. No content update. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From dfuchs at openjdk.java.net Tue Oct 13 14:25:18 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Tue, 13 Oct 2020 14:25:18 GMT Subject: Integrated: 8229867: Re-examine synchronization usages in http and https protocol handlers In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 11:22:09 GMT, Daniel Fuchs wrote: > Hi, > > This is a fix that upgrades the old HTTP and HTTPS legacy stack to use virtual-thread friendly locking instead of > synchronized monitors. > Most of the changes are mechanical - but there are still a numbers of subtle non-mechanical differences that are > outlined below: > 1. src/java.base/share/classes/sun/net/www/MessageHeader.java: > `MessageHeader::print(PrintStream)` => synchronization modified to not synchronize on this while printing > ( a snapshot of the data is taken before printing instead) > > 2. src/java.base/share/classes/sun/net/www/MeteredStream.java: > `MeteredStream::close` was missing synchronization: it is now protected by the lock > > 3. src/java.base/share/classes/sun/net/www/http/ChunkedOutputStream.java: > `ChunkedOutputStream` no longer extends `PrintStream` but extends `OutputStream` directly. > Extending `PrintStream` is problematic for virtual thread. After careful analysis, it appeared that > `ChunkedOutputStream` didn't really need to extend `PrintStream`. `ChunkedOutputStream` > is already wrapping a `PrintStream`. `ChunkedOutputStream` is never returned directly to user > code but is wrapped in another stream. `ChunkedOutputStream` completely reimplement and > reverse the flush logic implemented by its old super class`PrintStream` which leads me to believe > there was no real reason for it to extend `PrintStream` - except for being able to call its `checkError` > method - which can be done by using `instanceof ChunkedOutputStream` in the caller instead of > casting to PrintStream. > > 4. src/java.base/share/classes/sun/net/www/http/HttpClient.java: > Synchronization removed from `HttpClient::privilegedOpenServer` and replaced by an `assert`. > There is no need for a synchronized here as the method is only called from a method that already > holds the lock. > > 5. src/java.base/share/classes/sun/net/www/http/KeepAliveCache.java: > Synchronization removed from `KeepAliveCache::removeVector` and replaced by an `assert`. > This method is only called from methods already protected by the lock. > Also the method has been made private. > > 6. src/java.base/share/classes/sun/net/www/http/KeepAliveStream.java > `queuedForCleanup` is made volatile as it is read and written directly from outside the class > and without protection by the `KeepAliveCleanerEntry`. > Lock protection is also added to `close()`, which was missing it. > Some methods that have no lock protection and did not need it because only called from > within code blocks protected by the lock have aquired an `assert isLockHeldByCurrentThread();` > > 7. Concrete subclasses of `AuthenticationInfo` that provide an implementation for > `AuthenticationInfo::setHeaders(HttpURLConnection conn, HeaderParser p, String raw)` have > acquired an `assert conn.isLockheldByCurrentThread();` as the method should only be called > from within a lock-protected block in `s.n.w.p.h.HttpURLConnection` > > 8. src/java.base/share/classes/sun/net/www/protocol/http/HttpURLConnection.java > Several methods in this class have a acquired an `assert isLockheldByCurrentThread();` > to help track the fact that AuthenticationInfo::setHeaders is only called while > holding the lock. > Synchronization was also removed from some method that didn't need it because only > called from within code blocks protected by the lock: > `getOutputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `setCookieHeader()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `getInputStream0()`: synchronization removed and replaced by an `assert isLockheldByCurrentThread();` > `StreamingOutputStream`: small change to accomodate point 3. above (changes in ChunkedOutputStream). > > 9. src/java.base/share/classes/sun/net/www/protocol/http/NegotiateAuthentication.java.: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 10. src/java.base/unix/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. > > 11. src/java.base/windows/classes/sun/net/www/protocol/http/ntlm/NTLMAuthentication.java: > synchronization removed from `setHeaders` and replace by an assert. See point 7. above. This pull request has now been integrated. Changeset: 65393a09 Author: Daniel Fuchs URL: https://git.openjdk.java.net/jdk/commit/65393a09 Stats: 1126 lines in 18 files changed: 558 ins; 156 del; 412 mod 8229867: Re-examine synchronization usages in http and https protocol handlers Reviewed-by: chegar, alanb, michaelm ------------- PR: https://git.openjdk.java.net/jdk/pull/558 From lancea at openjdk.java.net Tue Oct 13 16:59:17 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Tue, 13 Oct 2020 16:59:17 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v4] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Wed, 7 Oct 2020 17:04:02 GMT, Lance Andersen wrote: >> I totally agree. What about using just the last sentence (as you've proposed) in the spec section and add the other to >> as @implNote? O you think the last sentence will be enough? > > I think we can just go with the last sentence/paragraph. Perhaps we can further simplify the paragraph/sentence with > something like: > The compressed entry size will be recalculated for compressed (DEFLATED) entries when ZipEntry::setCompressedSize has > not been explicitly called on the ZipEntry. > or > > The compressed (DEFLATED) entry size will be recalculated when ZipEntry::setCompressedSize has not been explicitly > called on the ZipEntry. I think the wording looks much better now ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Tue Oct 13 17:19:23 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Tue, 13 Oct 2020 17:19:23 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v4] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 13 Oct 2020 11:40:36 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev > excludes the unrelated changes brought in by the merge/rebase. The pull request contains one additional commit since > the last revision: > 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size test/jdk/java/util/zip/CopyZipFile.java line 149: > 147: os = new ByteArrayOutputStream(); > 148: zos = new ZipOutputStream(os); > 149: ZipFile zf = new ZipFile(ZIP_FILE); Any reason to not use try with resources here as well (and below) ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From jamil.j.nimeh at oracle.com Tue Oct 13 17:59:26 2020 From: jamil.j.nimeh at oracle.com (Jamil Nimeh) Date: Tue, 13 Oct 2020 10:59:26 -0700 Subject: RFR CSR: JDK-8254709 (Support for EdDSA signature scheme in JSSE) Message-ID: Hi Folks, I just put out the draft CSR for the RFE that adds EdDSA support in JSSE.? If anyone has some spare cycles to review this I'd appreciate it. https://bugs.openjdk.java.net/browse/JDK-8254709 Thanks, --Jamil From alanb at openjdk.java.net Tue Oct 13 18:53:23 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Tue, 13 Oct 2020 18:53:23 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v4] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 13 Oct 2020 11:40:36 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has updated the pull request with a new target base due to a merge or a rebase. The pull request now > contains one commit: > 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size src/java.base/share/classes/java/util/jar/JarOutputStream.java line 85: > 83: * ZipEntry#setCompressedSize(long)} method, then the compressed > 84: * size will be set to the actual compressed size after deflation. > 85: *

I'm happy with the wording. What you think about put a p tag before "The default compression method ..." so that it goes into its own paragraph? I'm asking because it's inconsistent to have the first paragraph include the details on the compression method and the details on the current time pushed into a second paragraph - if you generate the javadoc then you'll see what I mean. src/java.base/share/classes/java/util/zip/ZipEntry.java line 453: > 451: public void setCompressedSize(long csize) { > 452: this.csize = csize; > 453: this.csizeSet = true; Latest version of implementation changes look good. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Tue Oct 13 18:53:24 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Tue, 13 Oct 2020 18:53:24 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v4] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 13 Oct 2020 18:46:59 GMT, Alan Bateman wrote: >> Volker Simonis has updated the pull request with a new target base due to a merge or a rebase. The pull request now >> contains one commit: >> 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size > > src/java.base/share/classes/java/util/jar/JarOutputStream.java line 85: > >> 83: * ZipEntry#setCompressedSize(long)} method, then the compressed >> 84: * size will be set to the actual compressed size after deflation. >> 85: *

> > I'm happy with the wording. What you think about put a p tag before "The default compression method ..." so that it > goes into its own paragraph? I'm asking because it's inconsistent to have the first paragraph include the details on > the compression method and the details on the current time pushed into a second paragraph - if you generate the javadoc > then you'll see what I mean. I think that is a good idea to at the paragraph tag as you suggest ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Tue Oct 13 19:00:32 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Tue, 13 Oct 2020 19:00:32 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v5] In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/520/files - new: https://git.openjdk.java.net/jdk/pull/520/files/3edcf72e..0e2e130a Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=03-04 Stats: 115 lines in 1 file changed: 42 ins; 43 del; 30 mod Patch: https://git.openjdk.java.net/jdk/pull/520.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/520/head:pull/520 PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Tue Oct 13 19:07:33 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Tue, 13 Oct 2020 19:07:33 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v6] In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/520/files - new: https://git.openjdk.java.net/jdk/pull/520/files/0e2e130a..7f032e0f Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=04-05 Stats: 9 lines in 2 files changed: 3 ins; 0 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/520.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/520/head:pull/520 PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Tue Oct 13 19:07:37 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Tue, 13 Oct 2020 19:07:37 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v4] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 13 Oct 2020 17:16:21 GMT, Lance Andersen wrote: >> Volker Simonis has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev >> excludes the unrelated changes brought in by the merge/rebase. > > test/jdk/java/util/zip/CopyZipFile.java line 149: > >> 147: os = new ByteArrayOutputStream(); >> 148: zos = new ZipOutputStream(os); >> 149: ZipFile zf = new ZipFile(ZIP_FILE); > > Any reason to not use try with resources here as well (and below) Done ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Tue Oct 13 19:07:37 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Tue, 13 Oct 2020 19:07:37 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v4] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 13 Oct 2020 18:50:00 GMT, Lance Andersen wrote: >> src/java.base/share/classes/java/util/jar/JarOutputStream.java line 87: >> >>> 85: *

>>> 86: * The current time will be used if the entry has no set modification >>> 87: * time. >> >> I'm happy with the wording. What you think about put a p tag before "The default compression method ..." so that it >> goes into its own paragraph? I'm asking because it's inconsistent to have the first paragraph include the details on >> the compression method and the details on the current time pushed into a second paragraph - if you generate the javadoc >> then you'll see what I mean. > > I think that is a good idea to at the paragraph tag as you suggest Done ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Tue Oct 13 19:59:20 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Tue, 13 Oct 2020 19:59:20 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v6] In-Reply-To: <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> Message-ID: <75L2BpBchAkkPMo8-DtsWC48mN8N9wcdXfLH-e854L4=.bd404a73-6045-4fff-8143-5097041411b3@github.com> On Tue, 13 Oct 2020 19:07:33 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. test/jdk/java/util/zip/CopyZipFile.java line 104: > 102: // all these fields set to '-1'. > 103: InputStream is = new FileInputStream(ZIP_FILE); > 104: ZipInputStream zis = new ZipInputStream(is); Any reason not to include the the ZipInputStream and InputStream in the try with Resources and the ZipFile below? I know these are nits, but just curious or was it an over site? I think we are good otherwise ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Tue Oct 13 20:03:23 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Tue, 13 Oct 2020 20:03:23 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v6] In-Reply-To: <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> Message-ID: On Tue, 13 Oct 2020 19:07:33 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. test/jdk/java/util/zip/CopyZipFile.java line 140: > 138: // This means that all ZipEntry objects returned from ZipFile will have correct > 139: // settings for these fields. > 140: // I the compression level was different in the initial zip file (which we can't find typo I -> If ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Tue Oct 13 20:09:12 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Tue, 13 Oct 2020 20:09:12 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v6] In-Reply-To: <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> Message-ID: On Tue, 13 Oct 2020 19:07:33 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. test/jdk/java/util/zip/CopyZipFile.java line 188: > 186: if ("test1.txt".equals(entry.getName()) || "test2.txt".equals(entry.getName())) { > 187: throw new Exception( > 188: "Should throw for STORED files or files compressed with DEFAULT_COMPRESSION"); Perhaps clean up the message above "Should throw" to indicate what is being thrown ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From weijun at openjdk.java.net Wed Oct 14 03:51:23 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Wed, 14 Oct 2020 03:51:23 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: signing time, jarsigner -directsign, and digest algorithm check ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/322/files - new: https://git.openjdk.java.net/jdk/pull/322/files/ffaae532..734fd034 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=05-06 Stats: 53 lines in 5 files changed: 42 ins; 0 del; 11 mod Patch: https://git.openjdk.java.net/jdk/pull/322.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/322/head:pull/322 PR: https://git.openjdk.java.net/jdk/pull/322 From valeriep at openjdk.java.net Wed Oct 14 04:03:14 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 14 Oct 2020 04:03:14 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: On Tue, 13 Oct 2020 13:34:27 GMT, Weijun Wang wrote: >> Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: >> >> - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner >> >> - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature >> algorithms >> >> - A new JarSigner property "directsign" >> >> - Updating the jarsigner tool doc >> >> Major code changes: >> >> - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm >> there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. >> >> - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java >> >> - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId >> >> - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing >> >> - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms >> >> - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed > > Weijun Wang has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. src/java.base/share/classes/sun/security/x509/AlgorithmId.java line 113: > 111: } > 112: > 113: public AlgorithmId(ObjectIdentifier oid, DerValue params) Now that this is public, can we add a javadoc spec for it? src/java.base/share/classes/sun/security/util/SignatureUtil.java line 50: > 48: > 49: /** > 50: * Convent OID.1.2.3.4 or 1.2.3.4 to the matched stdName. Convent => Convert? matched => matching? Or maybe just "Match OID..... to the stdName" src/java.base/share/classes/sun/security/util/SignatureUtil.java line 53: > 51: * > 52: * @param algName input, could be in any form > 53: * @return the original name is it does not look like an OID, is => if? src/java.base/share/classes/sun/security/util/SignatureUtil.java line 54: > 52: * @param algName input, could be in any form > 53: * @return the original name is it does not look like an OID, > 54: * the matched name for an OID, or the OID itself Maybe use "the OID value"? The term "itself" reminds me of the specified argument somehow. Just nit. src/java.base/share/classes/sun/security/util/SignatureUtil.java line 94: > 92: * @return an AlgorithmParameterSpec object > 93: * @throws ProviderException > 94: */ Well, I am a bit unsure about your changes to this method. With your change, it returns default parameter spec (instead of null) when the specified AlgorithmParameters object is null. This may not be desirable for all cases? Existing callers would have to check for (params != null) before calling this method. The javadoc description also seems a bit strange with the to-be-converted AlgorithmParameters object being optional. Maybe add a separate method like `getParamSpecWithDefault` on top of this method or add a separate boolean argument `useDefault`? src/java.base/share/classes/sun/security/util/SignatureUtil.java line 164: > 162: } > 163: } else { > 164: paramSpec = getDefaultAlgorithmParameterSpec(sigName, null); Same comment here as for the getParamSpec(String, AlgorithmParameters) above. Given that there are two methods named getParamSpec(...), maybe add a boolean argument to indicate whether to return default value if the to-be-converted object is null? src/java.base/share/classes/sun/security/util/SignatureUtil.java line 290: > 288: * that must be initialized with a AlgorithmParameterSpec now. > 289: */ > 290: public static AlgorithmParameterSpec getDefaultAlgorithmParameterSpec( Maybe we can just use "getDefaultParamSpec" to match other methods' name in this class. Just a suggestion. src/java.base/share/classes/sun/security/util/SignatureUtil.java line 511: > 509: } > 510: > 511: // Same values for RSA and DSA Update the comment with "Return the default message digest algorithm with the same security strength as the specified IFC/FFC key size"? src/java.base/share/classes/sun/security/util/SignatureUtil.java line 499: > 497: } > 498: > 499: // Values from SP800-57 part 1 rev 4 tables 2 and 3 Update the comment with "Return the default message digest algorithm with the same security strength as the specified EC key size"? src/java.base/share/classes/sun/security/util/SignatureUtil.java line 367: > 365: * @param sigEngine the signature object > 366: * @param key the private key > 367: * @return the AlgorithmIA, not null IA => Id? src/java.base/share/classes/sun/security/util/SignatureUtil.java line 268: > 266: > 267: /** > 268: * Extracts the digest algorithm names from a signature names => name src/java.base/share/classes/sun/security/util/SignatureUtil.java line 205: > 203: try { > 204: sha512 = AlgorithmId.get(KnownOIDs.SHA_512.stdName()); > 205: shake256 = AlgorithmId.get(KnownOIDs.SHAKE256.stdName()); For these two, why not just do `new AlgorithmId(ObjectIdentifier.of(ko))` where ko = KnownOIDs.SHA_512, KnownOIDs.SHAKE256? More direct this way. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From valeriep at openjdk.java.net Wed Oct 14 04:07:19 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 14 Oct 2020 04:07:19 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Wed, 14 Oct 2020 03:51:23 GMT, Weijun Wang wrote: >> Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: >> >> - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner >> >> - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature >> algorithms >> >> - A new JarSigner property "directsign" >> >> - Updating the jarsigner tool doc >> >> Major code changes: >> >> - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm >> there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. >> >> - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java >> >> - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId >> >> - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing >> >> - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms >> >> - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed > > Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: > > signing time, jarsigner -directsign, and digest algorithm check src/java.base/share/classes/sun/security/util/KnownOIDs.java line 147: > 145: SHAKE128("2.16.840.1.101.3.4.2.11"), > 146: SHAKE256("2.16.840.1.101.3.4.2.12"), > 147: SHAKE256_LEN("2.16.840.1.101.3.4.2.18", "SHAKE256-LEN"), Can we move this down a little? The ordering within the section is based on the oid value. It's easier to check for unlisted/unsupported oid value this way. Why not also add SHAKE128_LEN? ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From alanb at openjdk.java.net Wed Oct 14 05:04:19 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Wed, 14 Oct 2020 05:04:19 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v6] In-Reply-To: <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> Message-ID: On Tue, 13 Oct 2020 19:07:33 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Wed Oct 14 10:54:30 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Wed, 14 Oct 2020 10:54:30 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v7] In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: <8NaRPfinIcIKT6GlY1mqQxGj43ZwwRYp6LYns504BeU=.c52e1ecc-5442-41e3-9e88-1b0be10bdf11@github.com> > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/520/files - new: https://git.openjdk.java.net/jdk/pull/520/files/7f032e0f..6b570c66 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=520&range=05-06 Stats: 13 lines in 1 file changed: 2 ins; 2 del; 9 mod Patch: https://git.openjdk.java.net/jdk/pull/520.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/520/head:pull/520 PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Wed Oct 14 10:54:32 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Wed, 14 Oct 2020 10:54:32 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v6] In-Reply-To: <75L2BpBchAkkPMo8-DtsWC48mN8N9wcdXfLH-e854L4=.bd404a73-6045-4fff-8143-5097041411b3@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> <75L2BpBchAkkPMo8-DtsWC48mN8N9wcdXfLH-e854L4=.bd404a73-6045-4fff-8143-5097041411b3@github.com> Message-ID: On Tue, 13 Oct 2020 19:56:50 GMT, Lance Andersen wrote: >> Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. > > test/jdk/java/util/zip/CopyZipFile.java line 104: > >> 102: // all these fields set to '-1'. >> 103: InputStream is = new FileInputStream(ZIP_FILE); >> 104: ZipInputStream zis = new ZipInputStream(is); > > Any reason not to include the the ZipInputStream and InputStream in the try with Resources and the ZipFile below? I > know these are nits, but just curious or was it an over site? > I think we are good otherwise No other reason except my laziness :) > test/jdk/java/util/zip/CopyZipFile.java line 140: > >> 138: // This means that all ZipEntry objects returned from ZipFile will have correct >> 139: // settings for these fields. >> 140: // I the compression level was different in the initial zip file (which we can't find > > typo I -> If Fixed > test/jdk/java/util/zip/CopyZipFile.java line 188: > >> 186: if ("test1.txt".equals(entry.getName()) || "test2.txt".equals(entry.getName())) { >> 187: throw new Exception( >> 188: "Should throw for STORED files or files compressed with DEFAULT_COMPRESSION"); > > Perhaps clean up the message above "Should throw" to indicate what is being thrown Thanks. That should actually red "Shouldn't...". Fixed that and also added ZipExcpetion as exception which shouldn't be thrown. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Wed Oct 14 10:54:32 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Wed, 14 Oct 2020 10:54:32 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v6] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <_XrGXAUZwdxVWjx_vSOg2LK-YzijZ7c46-ract7Znd0=.f4f7b55e-ea86-4b5f-8322-9311b0fc3a87@github.com> <75L2BpBchAkkPMo8-DtsWC48mN8N9wcdXfLH-e854L4=.bd404a73-6045-4fff-8143-5097041411b3@github.com> Message-ID: <5TcjFbuMieF40GRe46m9IY6YNSm8OwVWpVW7lIWJe7s=.3446c3e4-a520-41f1-b86c-9fe2dd55503b@github.com> On Wed, 14 Oct 2020 10:48:55 GMT, Volker Simonis wrote: >> test/jdk/java/util/zip/CopyZipFile.java line 104: >> >>> 102: // all these fields set to '-1'. >>> 103: InputStream is = new FileInputStream(ZIP_FILE); >>> 104: ZipInputStream zis = new ZipInputStream(is); >> >> Any reason not to include the the ZipInputStream and InputStream in the try with Resources and the ZipFile below? I >> know these are nits, but just curious or was it an over site? >> I think we are good otherwise > > No other reason except my laziness :) fixed now ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Wed Oct 14 14:56:18 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Wed, 14 Oct 2020 14:56:18 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v7] In-Reply-To: <8NaRPfinIcIKT6GlY1mqQxGj43ZwwRYp6LYns504BeU=.c52e1ecc-5442-41e3-9e88-1b0be10bdf11@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <8NaRPfinIcIKT6GlY1mqQxGj43ZwwRYp6LYns504BeU=.c52e1ecc-5442-41e3-9e88-1b0be10bdf11@github.com> Message-ID: On Wed, 14 Oct 2020 10:54:30 GMT, Volker Simonis wrote: >> ### Summary >> >> Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid >> entry compressed size"`. >> ### Motivation >> >> In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, >> `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a >> `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as >> follows: >> ZipEntry entry; >> ZipInputStream zis = new ZipInputStream(...); >> ZipOutputStream zos = new ZipOutputStream(...); >> while((entry = zis.getNextEntry()) != null) { >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> The problem with this code is that the zip file format does not record the compression level used for deflation in its >> entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the >> compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the >> receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: >> java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) >> >> The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at >> least resetting of the compressed size field. E.g.: >> while((entry = zis.getNextEntry()) != null) { >> ZipEntry newEntry = new ZipEntry(entry.getName()); >> zos.putNextEntry(newEntry); >> zis.transferTo(zos); >> } >> or: >> while((entry = zis.getNextEntry()) != null) { >> entry.setCompressedSize(-1); >> zos.putNextEntry(entry); >> zis.transferTo(zos); >> } >> Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described >> before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives >> ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when >> doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the >> latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see >> [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. >> [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has >> clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. >> However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with >> OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting >> `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for >> selected services in production and the only reason why we haven't enabled them by default until now is the problem >> I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because >> their compression ratio is slightly different from that of the default zlib library. This can easily trigger a >> `ZipException` even if an application is not using a different compression levels but just a zip file created with >> another zlib version. I'd therefore like to propose the following workaround for the wrong >> `ZipOutputStream.putNextEntry()` usage in user code: >> - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling >> `ZipEntry.setCompressedSize()`. >> >> - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the >> problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when >> reading that entry from a `ZipFile` or `ZipInputStream`. >> >> >> ### Technical Details >> >> A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed >> specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an >> optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed >> size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is >> created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the >> corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory >> File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate >> reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history >> when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing >> to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will >> initially only contain the information from the LFH. Only after the next entry was read (or after >> `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the >> Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only >> queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, >> `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and >> CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore >> the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, >> this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described >> before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by >> default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and >> CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with >> `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and >> inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a >> second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because >> the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only >> straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by >> copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or >> less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly >> set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such >> files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files >> created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the >> implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip >> archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it >> will give us the freedom to use whatever zip implementation we like :) [1]: >> https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 >> [2]: >> https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 >> [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: >> https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: >> https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) > > Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. The changes look good. Thank you for the updates. I have kicked off mach5 jdk-tier1,jdk-tier2,jdk-tier3 runs across all of the platforms. ------------- Marked as reviewed by lancea (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Wed Oct 14 14:56:19 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Wed, 14 Oct 2020 14:56:19 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v2] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <6RMxyurj4wb-oTldhToZhDebkRlbFrSTPzAWnhcni0Q=.457787d6-d1fe-455d-9ade-03e01c099db9@github.com> Message-ID: On Mon, 12 Oct 2020 11:46:31 GMT, Volker Simonis wrote: >> I don't believe we discuss/reference the data descriptor for a Zip entry (outside of the PKWare Zip specification) so I >> am not sure we should reference it in the javadoc. >> Placing the sentence after "The default compression method will be used if no compression method was specified for the >> entry" makes sense > > Thanks for your input. I've tried to somehow merge both suggestions :) > > Did you had a chance to look at the CSR? I think somebody has to review it before I can move it to "Finalized". > > Thank you and best regards, > Volker I have added myself as a reviewer based on the updates made by you and Alan ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From lancea at openjdk.java.net Wed Oct 14 16:34:17 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Wed, 14 Oct 2020 16:34:17 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v7] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <8NaRPfinIcIKT6GlY1mqQxGj43ZwwRYp6LYns504BeU=.c52e1ecc-5442-41e3-9e88-1b0be10bdf11@github.com> Message-ID: On Wed, 14 Oct 2020 14:52:29 GMT, Lance Andersen wrote: >> Volker Simonis has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. > > The changes look good. Thank you for the updates. I have kicked off mach5 jdk-tier1,jdk-tier2,jdk-tier3 runs across > all of the platforms. Mach5 run is clean :-) ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From simonis at openjdk.java.net Wed Oct 14 17:56:20 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Wed, 14 Oct 2020 17:56:20 GMT Subject: RFR: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size [v7] In-Reply-To: References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> <8NaRPfinIcIKT6GlY1mqQxGj43ZwwRYp6LYns504BeU=.c52e1ecc-5442-41e3-9e88-1b0be10bdf11@github.com> Message-ID: On Wed, 14 Oct 2020 16:31:32 GMT, Lance Andersen wrote: > Mach5 run is clean :-) Thanks a lot Lance. Just waiting for the CSR to get approved now. ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From valeriep at openjdk.java.net Wed Oct 14 19:03:13 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 14 Oct 2020 19:03:13 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Wed, 14 Oct 2020 03:51:23 GMT, Weijun Wang wrote: >> Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: >> >> - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner >> >> - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature >> algorithms >> >> - A new JarSigner property "directsign" >> >> - Updating the jarsigner tool doc >> >> Major code changes: >> >> - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm >> there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. >> >> - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java >> >> - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId >> >> - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing >> >> - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms >> >> - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed > > Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: > > signing time, jarsigner -directsign, and digest algorithm check src/java.base/share/classes/sun/security/util/SignatureUtil.java line 210: > 208: new DerValue((byte) 2, new byte[]{2, 0})); // int 512 > 209: } catch (IOException | NoSuchAlgorithmException e) { > 210: throw new AssertionError("Shoudl not happen", e); Shoudl => Should src/java.base/share/classes/sun/security/util/SignatureUtil.java line 215: > 213: } > 214: /** > 215: * Determines the digestEncryptionAlgorithmId in PKCS& SignerInfo. &=>7 src/java.base/share/classes/sun/security/util/SignatureUtil.java line 217: > 215: * Determines the digestEncryptionAlgorithmId in PKCS& SignerInfo. > 216: * > 217: * @param signer Signature object that tells you RSASA-PSS params RSASA=>RSASSA src/java.base/share/classes/sun/security/util/SignatureUtil.java line 221: > 219: * @param privateKey key tells you EdDSA params > 220: * @param directsign Ed448 uses different digest algs depending on this > 221: * @return the digest alg alg => algid src/java.base/share/classes/sun/security/util/SignatureUtil.java line 218: > 216: * > 217: * @param signer Signature object that tells you RSASA-PSS params > 218: * @param sigalg Signature algorithm tells you who with who who with who? ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From sean.mullan at oracle.com Wed Oct 14 21:08:11 2020 From: sean.mullan at oracle.com (Sean Mullan) Date: Wed, 14 Oct 2020 17:08:11 -0400 Subject: RFR CSR: JDK-8254709 (Support for EdDSA signature scheme in JSSE) In-Reply-To: References: Message-ID: <1f283a72-5b0b-6285-08cd-1d6e5f6afa18@oracle.com> In the Summary and Solution sections, can you be more specific as to what TLS versions will be supported? Can you also show what the order of signature schemes is before and after the change, for each TLS version? I think this would make it more clear about what the priority of the new schemes is. Thanks, Sean On 10/13/20 1:59 PM, Jamil Nimeh wrote: > Hi Folks, > > I just put out the draft CSR for the RFE that adds EdDSA support in > JSSE.? If anyone has some spare cycles to review this I'd appreciate it. > > https://bugs.openjdk.java.net/browse/JDK-8254709 > > Thanks, > > --Jamil > From simonis at openjdk.java.net Thu Oct 15 09:22:21 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Thu, 15 Oct 2020 09:22:21 GMT Subject: Integrated: 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size In-Reply-To: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> References: <0nq5oFKOj6AZqCW6jF268-6oKRI6UkS6TRIHo-_5vKw=.d13c7016-4473-4b20-9117-100b368d3716@github.com> Message-ID: On Tue, 6 Oct 2020 10:02:09 GMT, Volker Simonis wrote: > ### Summary > > Work around wrong usage of `ZipOutputStream.putNextEntry()` in user code which can lead to the `ZipException "invalid > entry compressed size"`. > ### Motivation > > In general it is not safe to directly write a ZipEntry obtained from `ZipInputStream.getNextEntry()`, > `ZipFile.entries()`, `ZipFile.getEntry()` or `ZipFile.stream()` with `ZipOutputStream.putNextEntry()` to a > `ZipOutputStream` and then read the entries data from the `ZipInputStream` and write it to the `ZipOutputStream` as > follows: > ZipEntry entry; > ZipInputStream zis = new ZipInputStream(...); > ZipOutputStream zos = new ZipOutputStream(...); > while((entry = zis.getNextEntry()) != null) { > zos.putNextEntry(entry); > zis.transferTo(zos); > } > The problem with this code is that the zip file format does not record the compression level used for deflation in its > entries. In general, it doesn't even mandate a predefined compression ratio per compression level. Therefore the > compressed size recorded in a `ZipEntry` read from a zip file might differ from the new compressed size produced by the > receiving `ZipOutputStream`. Such a difference will result in a `ZipException` with the following message: > java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes) > > The correct way of copying all entries from one zip file into another requires the creation of a new `ZipEntry` or at > least resetting of the compressed size field. E.g.: > while((entry = zis.getNextEntry()) != null) { > ZipEntry newEntry = new ZipEntry(entry.getName()); > zos.putNextEntry(newEntry); > zis.transferTo(zos); > } > or: > while((entry = zis.getNextEntry()) != null) { > entry.setCompressedSize(-1); > zos.putNextEntry(entry); > zis.transferTo(zos); > } > Unfortunately, there's a lot of user code out there which gets this wrong and uses the bad coding pattern described > before. Searching for `"java.util.zip.ZipException: invalid entry compressed size (expected 12 but got 7 bytes)"` gives > ~2500 hits (~100 on StackOverflow). It's also no hard to find plenty of instances of this anti-pattern on GitHub when > doing a code search for `ZipEntry` and `putNextEntry()`. E.g. [Gradle 4.x wrapper task][1] is affected as well as the > latest version of the [mockableAndroidJar task][2]. I've recently fixed two occurrences of this pattern in OpenJDK (see > [JDK-8240333][3] and [JDK-8240235][4]) but there still exist more of them (e.g. > [`test/jdk/java/util/zip/ZipFile/CopyJar.java`][5] which is there since 1999 :). ### Description So while this has > clearly been a problem before, it apparently wasn't painful enough to trigger any action from the side of the JDK. > However, recently quite some zlib forks with [superior deflate/inflate performance have evolved][6]. Using them with > OpenJDK is quite straight-forward: one just has to configure the alternative implementations by setting > `LD_LIBRARY_PATH` or `LD_PRELOAD` correspondingly. We've seen big saving by using these new zlib implementations for > selected services in production and the only reason why we haven't enabled them by default until now is the problem > I've just described. The reason why these new libraries uncover the described anti-pattern much more often is because > their compression ratio is slightly different from that of the default zlib library. This can easily trigger a > `ZipException` even if an application is not using a different compression levels but just a zip file created with > another zlib version. I'd therefore like to propose the following workaround for the wrong > `ZipOutputStream.putNextEntry()` usage in user code: > - ignore the compressed size if it was implicitly determined from the zip file and not explicitly set by calling > `ZipEntry.setCompressedSize()`. > > - Change the API-documentation of `ZipOutputStream.putNextEntry()` and `JarOutputStream.putNextEntry()` to explain the > problem and why `putNextEntry()` will ignore the compressed size of a `ZipEntry` if that was set implicitely when > reading that entry from a `ZipFile` or `ZipInputStream`. > > > ### Technical Details > > A zip file consists of a stream of File Entries followed by a Central Directory (see [here for a more detailed > specification][7]). Each File Entry is composed of a Local File Header (LFH) followed by the compressed Data and an > optional Data Descriptor. The LFH contains the File Name and among other attributes the Compressed and Uncompressed > size and CRC of the Data. In the case where the latter three attributes are not available at the time when the LFH is > created, this fact will be recorded in a flag of the LFH and will trigger the creation of a Data Descriptor with the > corresponding information right after the Data section. Finally, the Central Directory contains one Central Directory > File Header (CDFH) for each entry of the zip archive. The CDFH is an extended version of the LFH and the ultimate > reference for the contents of the zip archive. The redundancy between LFH and CDFH is a tribute to zip's long history > when it was used to store archives on multiple floppy discs and the CDFH allowed to update the archive by only writing > to the last disc which contained the Central Directory. `ZipEntries` read with `ZipInputStream.getNextEntry()` will > initially only contain the information from the LFH. Only after the next entry was read (or after > `ZipInputStream.closeEntry()` was called explicitly), will the previously read entry be updated with the data from the > Data Descriptor. `ZipInputStream` doesn't inspect the Central Directory at all. On the other hand, `ZipFile` only > queries the Central Directory for `ZipEntry` information so all `ZipEntries` returned by `ZipFile.entries()`, > `ZipFile.getEntry()` and `ZipFile.stream()` will always instantly contain the full Compressed and Uncompressed Size and > CRC information for each entry independently of the LFH contents. ### Risks and Assumptions If we choose to ignore > the implicitly recorded compressed size in a `ZipEntry` read from a zip file when writing it to a `ZipOutputStream`, > this will lead to zip files with incomplete information in the LFH and an additional Data Descriptor as described > before. However, the result is still fully compatible to the zip file specification. It's also not unusual, because by > default all new zip files created with `ZipOutputStream` will contain LFHs without Compressed and Uncompressed Size and > CRC information and an additional Data Descriptor. Theoretically it is possible to create new zip files with > `ZipOutputStream` class and Compressed and Uncompressed Size and CRC information in the LFH but that's complex and > inefficient because it requires two steps. A first step to determine the crc and compressed size of the data and a > second step to actually write the data to the `ZipOutputStream` (which will compress it a second time). This is because > the current API offers no possibility to write already compressed data to a `ZipOutputStream`. Consequently, the only > straight-forward way of creating zip files from Java which have all the data in the LFH and no Data Descriptor is by > copying `ZipEntries` from an existing zip file with the buggy method described before. This incidentally worked more or > less reliable for a long time but breaks miserably when using different zlib implementations. Ignoring the implicitly > set compressed size of `ZipEntries` can easily fix this problem. I'm not aware of any tool which can not handle such > files and if it exists it would have problems with the majority of Java created zip files anyway (e.g. all jar-files > created with the jar tool have zip entries with incomplete LFH data and additional Data Descriptor). Ignoring the > implicitly set compressed size of `ZipEntries` has no measurable performance impact and will increase the size of zip > archives which used to have the complete file information in the LFH before by 16 bytes per entry. On the other hand it > will give us the freedom to use whatever zip implementation we like :) [1]: > https://github.com/gradle/gradle/blob/e76905e3a/subprojects/build-init/src/main/java/org/gradle/api/tasks/wrapper/Wrapper.java#L152-L158 > [2]: > https://android.googlesource.com/platform/tools/base/+/refs/heads/master/build-system/builder/src/main/java/com/android/builder/testing/MockableJarGenerator.java#86 > [3]: https://bugs.openjdk.java.net/browse/JDK-8240333 [4]: https://bugs.openjdk.java.net/browse/JDK-8240235 [5]: > https://github.com/openjdk/jdk/blob/master/test/jdk/java/util/zip/ZipFile/CopyJar.java [6]: > https://github.com/simonis/zlib-bench/blob/master/Results.md [7]: https://en.wikipedia.org/wiki/Zip_(file_format) This pull request has now been integrated. Changeset: 60159cff Author: Volker Simonis URL: https://git.openjdk.java.net/jdk/commit/60159cff Stats: 231 lines in 4 files changed: 223 ins; 0 del; 8 mod 8253952: Refine ZipOutputStream.putNextEntry() to recalculate ZipEntry's compressed size Reviewed-by: lancea, alanb ------------- PR: https://git.openjdk.java.net/jdk/pull/520 From mcimadamore at openjdk.java.net Thu Oct 15 16:20:25 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 15 Oct 2020 16:20:25 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v2] In-Reply-To: References: Message-ID: <8CkvfiM_PR0iDEo-tgUNRXWlG-qDwEDN2Qd15pbgQ0E=.353d8045-3593-4c72-9d81-c290b43ca89c@github.com> > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and > associated pull request [3]). > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate > JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the > writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be > used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, > I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be > periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as > possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you > see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to > Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library > by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory > layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native > function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes > a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a > `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, > and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which > acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures > (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout > attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take > place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and > back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than > allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to > use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a > performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing > native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For > instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, > in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is > a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as > it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes > associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library > loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to > JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes > are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing > some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request > into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native > scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are > implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support > the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of > some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale > behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a > Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function > (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding > native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward > fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on > Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that > kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no > longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a > floating point value, or an integral value; this knowledge is required because floating points are passed in different > registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which > contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this > attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive > operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such > bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the > main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` > what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the > `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the > various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed > below: > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see > `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating > a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The > buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in > their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This > is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is > some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, > we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above > (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer > allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of > bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), > then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an > intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to > add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond > what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more > readings on the internals of the foreign linker support, please refer to [5]. > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) > which aim at testing the linker from the perspective of code that clients could write. But we also have deeper > combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI > implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s > for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the > linker machinery as a black box and verify that the support works by checking that the native call returned the results > we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also > mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing > on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - > https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - > https://git.openjdk.java.net/jdk/pull/548 [4] - > https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - > http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 19 commits: - Merge branch 'master' into 8254231_linker - Fix tests - Fix more whitespaces - Fix whitespaces - Remove rejected file - More updates - Add new files - Merge with master - Merge branch 'master' into 8254162 - Remove spurious check on MemoryScope::confineTo Added tests to make sure no spurious exception is thrown when: * handing off a segment from A to A * sharing an already shared segment - Merge branch 'master' into 8254162 - ... and 9 more: https://git.openjdk.java.net/jdk/compare/3c2f5e08...ad8bee12 ------------- Changes: https://git.openjdk.java.net/jdk/pull/634/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=01 Stats: 84781 lines in 279 files changed: 72658 ins; 10861 del; 1262 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 15 16:32:03 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 15 Oct 2020 16:32:03 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v3] In-Reply-To: References: Message-ID: <4w2EB316f1EZC9b-zMmHIa1xqQL6Jw0-Vif_CXzkDS4=.16eaa7d6-f59b-4622-8b34-b9df57d63f45@github.com> > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and > associated pull request [3]). > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate > JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the > writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be > used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, > I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be > periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as > possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you > see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to > Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library > by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory > layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native > function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes > a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a > `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, > and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which > acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures > (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout > attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take > place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and > back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than > allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to > use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a > performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing > native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For > instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, > in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is > a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as > it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes > associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library > loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to > JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes > are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing > some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request > into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native > scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are > implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support > the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of > some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale > behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a > Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function > (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding > native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward > fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on > Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that > kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no > longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a > floating point value, or an integral value; this knowledge is required because floating points are passed in different > registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which > contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this > attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive > operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such > bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the > main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` > what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the > `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the > various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed > below: > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see > `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating > a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The > buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in > their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This > is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is > some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, > we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above > (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer > allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of > bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), > then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an > intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to > add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond > what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more > readings on the internals of the foreign linker support, please refer to [5]. > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) > which aim at testing the linker from the perspective of code that clients could write. But we also have deeper > combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI > implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s > for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the > linker machinery as a black box and verify that the support works by checking that the native call returned the results > we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also > mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing > on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - > https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - > https://git.openjdk.java.net/jdk/pull/548 [4] - > https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - > http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Re-add erroneously removed files ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/634/files - new: https://git.openjdk.java.net/jdk/pull/634/files/ad8bee12..2184831e Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=01-02 Stats: 9218 lines in 14 files changed: 9218 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 15 17:08:28 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 15 Oct 2020 17:08:28 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: References: Message-ID: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and > associated pull request [3]). > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate > JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the > writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be > used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, > I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be > periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as > possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you > see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to > Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library > by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory > layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native > function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes > a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a > `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, > and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which > acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures > (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout > attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take > place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and > back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than > allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to > use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a > performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing > native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For > instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, > in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is > a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as > it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes > associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library > loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to > JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes > are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing > some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request > into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native > scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are > implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support > the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of > some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale > behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a > Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function > (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding > native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward > fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on > Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that > kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no > longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a > floating point value, or an integral value; this knowledge is required because floating points are passed in different > registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which > contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this > attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive > operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such > bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the > main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` > what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the > `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the > various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed > below: > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see > `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating > a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The > buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in > their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This > is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is > some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, > we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above > (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer > allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of > bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), > then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an > intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to > add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond > what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more > readings on the internals of the foreign linker support, please refer to [5]. > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) > which aim at testing the linker from the perspective of code that clients could write. But we also have deeper > combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI > implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s > for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the > linker machinery as a black box and verify that the support works by checking that the native call returned the results > we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also > mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing > on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - > https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - > https://git.openjdk.java.net/jdk/pull/548 [4] - > https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - > http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Re-add file erroneously deleted (detected as rename) ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/634/files - new: https://git.openjdk.java.net/jdk/pull/634/files/2184831e..830c5cea Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=02-03 Stats: 35 lines in 1 file changed: 35 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From vromero at openjdk.java.net Thu Oct 15 18:18:10 2020 From: vromero at openjdk.java.net (Vicente Romero) Date: Thu, 15 Oct 2020 18:18:10 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Tue, 13 Oct 2020 13:29:39 GMT, Weijun Wang wrote: >> Add support for [RFC 6211: Cryptographic Message Syntax (CMS) Algorithm Identifier Protection >> Attribute](https://tools.ietf.org/html/rfc6211) to protect against algorithm substitution attacks. This attribute is >> signed and it contains copies of digestAlgorithm and signatureAlgorithm which are unprotected in SignerInfo. Before >> this enhancement, the two algorithms can be implied from the signature itself (i.e. if you change any of them the >> signature size would not match or the key will not decrypt). However, with the introduction of RSASSA-PSS, the >> signature algorithm can be modified and it still looks like the signature is valid. This particular case is [described >> in the RFC](https://tools.ietf.org/html/rfc6211#page-5): >> signatureAlgorithm has been protected by implication in the past. >> The use of an RSA public key implied that the RSA v1.5 signature >> algorithm was being used. The hash algorithm and this fact could >> be checked by the internal padding defined. This is no longer >> true with the addition of the RSA-PSS signature algorithms. > > A force push to fix the RFC number typo in the latest commit. No content update. this one has nothing to do with javac so the `compiler` label should be removed ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From bradford.wetmore at oracle.com Thu Oct 15 18:19:10 2020 From: bradford.wetmore at oracle.com (Bradford Wetmore) Date: Thu, 15 Oct 2020 11:19:10 -0700 Subject: RFR CSR: JDK-8254709 (Support for EdDSA signature scheme in JSSE) In-Reply-To: <1f283a72-5b0b-6285-08cd-1d6e5f6afa18@oracle.com> References: <1f283a72-5b0b-6285-08cd-1d6e5f6afa18@oracle.com> Message-ID: <1fc92812-1871-f1f7-b3ec-f1f243b4c852@oracle.com> Like XDH, these... -> Like XDH (i.e. x25519/x448), these... You might provide a link to the original XDH JEP (JDK-8181595) and later CSR (JDK-8224520) and/or JDK-8171279. and for use in automatic certificate selection for certificate messages. -> and for use in certificate selection for certificate messages. KeyManagers aren't required to be "automatic." e.g. the old plugin used to popup a dialog box that waited for the user to select which key/cert to use. What specific key values will be used on the KeyManager.*ClientAlias*()? "signature_algorithms and signature_algorithms_cert" I was originally going to suggest trying to describe these functions and whether they appear in TLSv1.2/1.3 but might be a little hard to explain, but might not be worth it. Your call if so. Otherwise, this looks good. I've added myself as reviewer. Brad On 10/14/2020 2:08 PM, Sean Mullan wrote: > In the Summary and Solution sections, can you be more specific as to > what TLS versions will be supported? > > Can you also show what the order of signature schemes is before and > after the change, for each TLS version? I think this would make it more > clear about what the priority of the new schemes is. > > Thanks, > Sean > > On 10/13/20 1:59 PM, Jamil Nimeh wrote: >> Hi Folks, >> >> I just put out the draft CSR for the RFE that adds EdDSA support in >> JSSE.? If anyone has some spare cycles to review this I'd appreciate it. >> >> https://bugs.openjdk.java.net/browse/JDK-8254709 >> >> Thanks, >> >> --Jamil >> From jvernee at openjdk.java.net Thu Oct 15 18:24:15 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Thu, 15 Oct 2020 18:24:15 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) In-Reply-To: References: Message-ID: <9f-iShxyaRbjn74wlAqBPH3ycRWisVwlWuB81Kv8Q6A=.be903417-dda9-434a-96a4-d9ea6a1e1737@github.com> On Tue, 13 Oct 2020 13:08:14 GMT, Maurizio Cimadamore wrote: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and > associated pull request [3]). > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate > JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the > writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be > used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, > I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be > periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as > possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you > see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to > Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library > by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory > layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native > function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes > a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a > `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, > and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which > acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures > (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout > attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take > place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and > back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than > allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to > use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a > performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing > native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For > instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, > in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is > a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as > it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes > associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library > loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to > JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes > are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing > some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request > into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native > scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are > implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support > the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of > some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale > behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a > Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function > (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding > native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward > fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on > Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that > kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no > longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a > floating point value, or an integral value; this knowledge is required because floating points are passed in different > registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which > contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this > attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive > operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such > bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the > main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` > what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the > `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the > various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed > below: > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see > `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating > a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The > buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in > their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This > is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is > some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, > we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above > (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer > allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of > bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), > then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an > intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to > add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond > what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more > readings on the internals of the foreign linker support, please refer to [5]. > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) > which aim at testing the linker from the perspective of code that clients could write. But we also have deeper > combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI > implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s > for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the > linker machinery as a black box and verify that the support works by checking that the native call returned the results > we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also > mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing > on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - > https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - > https://git.openjdk.java.net/jdk/pull/548 [4] - > https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - > http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Here is the updated partial webrev against https://github.com/openjdk/jdk/pull/548 : http://cr.openjdk.java.net/~jvernee/linker_rfr/v2/ ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From weijun at openjdk.java.net Thu Oct 15 18:34:12 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 15 Oct 2020 18:34:12 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Thu, 15 Oct 2020 18:15:35 GMT, Vicente Romero wrote: > this one has nothing to do with javac so the `compiler` label should be removed @vicente-romero-oracle Sorry for the noise, I should have removed it earlier. All files in the `jdk.jartool` module are under `compiler` in https://github.com/openjdk/skara/blob/master/config/mailinglist/rules/jdk.json#L120. You can make it more specific. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From coleenp at openjdk.java.net Thu Oct 15 23:18:14 2020 From: coleenp at openjdk.java.net (Coleen Phillimore) Date: Thu, 15 Oct 2020 23:18:14 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> Message-ID: <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> On Thu, 15 Oct 2020 17:08:28 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the first incubation round of the foreign linker access API incubation >> (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and >> associated pull request [3]). >> The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate >> JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the >> writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be >> used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, >> I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be >> periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as >> possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you >> see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to >> Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio >> Webrev: >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev >> >> Javadoc: >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html >> >> Specdiff (relative to [3]): >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254232 >> >> >> >> ### API Changes >> >> The API changes are actually rather slim: >> >> * `LibraryLookup` >> * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library >> by name, or absolute path, and then lookup symbols on that library. >> * `FunctionDescriptor` >> * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory >> layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native >> function. >> * `CLinker` >> * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes >> a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a >> `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, >> and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which >> acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. >> * This class also contains the various layout constants that should be used by clients when describing native signatures >> (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout >> attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take >> place. >> * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and >> back. >> * `NativeScope` >> * This is an helper class which allows clients to group together logically related allocations; that is, rather than >> allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to >> use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a >> performance boost, since not all allocation requests will be turned into `malloc` calls. >> * `MemorySegment` >> * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing >> native scope. >> >> ### Safety >> >> The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For >> instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, >> in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is >> a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as >> it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes >> associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library >> loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to >> JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes >> are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing >> some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request >> into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native >> scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are >> implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support >> the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of >> some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale >> behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a >> Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function >> (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding >> native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward >> fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on >> Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that >> kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no >> longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a >> floating point value, or an integral value; this knowledge is required because floating points are passed in different >> registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which >> contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this >> attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive >> operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such >> bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the >> main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` >> what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the >> `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the >> various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed >> below: >> * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see >> `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating >> a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The >> buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in >> their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This >> is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is >> some extra allocation which takes place. >> >> * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, >> we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). >> >> * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above >> (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer >> allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of >> bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), >> then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an >> intermediate buffer. This gives us back performances that are on par with JNI. >> >> For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to >> add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond >> what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more >> readings on the internals of the foreign linker support, please refer to [5]. >> #### Test changes >> >> Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) >> which aim at testing the linker from the perspective of code that clients could write. But we also have deeper >> combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI >> implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s >> for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the >> linker machinery as a black box and verify that the support works by checking that the native call returned the results >> we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also >> mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing >> on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - >> https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - >> https://git.openjdk.java.net/jdk/pull/548 [4] - >> https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - >> http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Re-add file erroneously deleted (detected as rename) I looked through some Hotspot runtime code and that looks ok. I saw a couple of strange things on my way through the code. See comments. src/hotspot/cpu/x86/foreign_globals_x86.cpp line 2: > 1: /* > 2: * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. Copyright should be 2020. All the new files should have 2020 as the copyright, a bunch don't. src/hotspot/cpu/x86/foreign_globals_x86.cpp line 56: > 54: } > 55: > 56: const ABIDescriptor parseABIDescriptor(JNIEnv* env, jobject jabi) { I don't know if you care about performance but of these env->calls transition into the VM and back out again. You should prefix all the code that comes from java to native with JNI_ENTRY and just use native JVM code to implement these. src/hotspot/cpu/x86/foreign_globals_x86.hpp line 32: > 30: #define __ _masm-> > 31: > 32: struct VectorRegister { Why are these structs and not classes? src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp line 3885: > 3883: > 3884: __ flush(); > 3885: } I think as a future RFE we should refactor this function and generate_native_wrapper since they're similar (this is nicer to read). If I can remove is_critical_native code they will be more similar. ------------- Changes requested by coleenp (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/634 From valeriep at openjdk.java.net Fri Oct 16 00:02:15 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 16 Oct 2020 00:02:15 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Wed, 14 Oct 2020 03:51:23 GMT, Weijun Wang wrote: >> Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: >> >> - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner >> >> - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature >> algorithms >> >> - A new JarSigner property "directsign" >> >> - Updating the jarsigner tool doc >> >> Major code changes: >> >> - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm >> there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. >> >> - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java >> >> - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId >> >> - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing >> >> - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms >> >> - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed > > Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: > > signing time, jarsigner -directsign, and digest algorithm check src/java.base/share/classes/sun/security/pkcs/PKCS7.java line 172: > 170: throws IOException > 171: { > 172: ContentInfo block = new ContentInfo(derin, oldStyle); With this change, i.e. using a local variable instead of setting the field 'contentInfo', the 'contentInfo' field seems to left unset when contentType equals to ContentInfo.NETSCAPE_CERT_SEQUENCE_OID? src/java.base/share/classes/sun/security/pkcs/PKCS7.java line 818: > 816: DerOutputStream derSigAlg = new DerOutputStream(); > 817: sigAlgID.derEncode(derSigAlg); > 818: derAlgs.writeImplicit((byte)0xA1, derSigAlg); Are you sure that this context specific tag value is implicit? In RFC 6211, some other ASN.1 definition uses IMPLICIT keyword after the [x] which seems to suggest that the default is explicit unless specified. Besides, the layman's guide sec2.3 also states "The keyword [class number] alone is the same as explicit tagging, except when the "module" in which the ASN.1 type is defined has implicit tagging by default." So, it seems that explicit tagging should be the default? src/java.base/share/classes/sun/security/pkcs/SignerInfo.java line 508: > 506: * name and the encryption algorithm name inside a PKCS7 SignerInfo. > 507: * > 508: * For old style PKCS7 files where we use RSA, DSA, EC asencAlgId asencAlgId => as encAlgId src/java.base/share/classes/sun/security/pkcs/SignerInfo.java line 549: > 547: return encAlg; > 548: default: > 549: String digAlg = digAlgId.getName().replace("-", ""); This may be incorrect if the digest algorithm is in the SHA3 family. Maybe we should check and apply this conversion only when digest algorithm starts with "SHA-". src/jdk.jartool/share/classes/jdk/security/jarsigner/JarSigner.java line 858: > 856: byte[] content = baos.toByteArray(); > 857: > 858: // Use new method is directSign is false or it's a modern is => if ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 01:44:19 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 01:44:19 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: On Tue, 13 Oct 2020 23:50:05 GMT, Valerie Peng wrote: >> Weijun Wang has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. > > src/java.base/share/classes/sun/security/util/SignatureUtil.java line 94: > >> 92: * @return an AlgorithmParameterSpec object >> 93: * @throws ProviderException >> 94: */ > > Well, I am a bit unsure about your changes to this method. With your change, it returns default parameter spec (instead > of null) when the specified AlgorithmParameters object is null. This may not be desirable for all cases? Existing > callers would have to check for (params != null) before calling this method. The javadoc description also seems a bit > strange with the to-be-converted AlgorithmParameters object being optional. Maybe add a separate method like > `getParamSpecWithDefault` on top of this method or add a separate boolean argument `useDefault`? I cannot remember why I need to return a default. The only default we current have is for RSASSA-PSS, and in all RSASSA-PSS AlgorithmId for signature I see there is always the params. (When it's for a key the params can be missing). All 3 callers of this method is on a signature AlgorithmId so the params should not be null. I'll remove the default return value and do more testing. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From valeriep at openjdk.java.net Fri Oct 16 01:44:20 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 16 Oct 2020 01:44:20 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: On Tue, 13 Oct 2020 13:34:27 GMT, Weijun Wang wrote: >> Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: >> >> - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner >> >> - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature >> algorithms >> >> - A new JarSigner property "directsign" >> >> - Updating the jarsigner tool doc >> >> Major code changes: >> >> - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm >> there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. >> >> - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java >> >> - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId >> >> - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing >> >> - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms >> >> - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed > > Weijun Wang has refreshed the contents of this pull request, and previous commits have been removed. The incremental > views will show differences compared to the previous content of the PR. test/jdk/jdk/security/jarsigner/Spec.java line 128: > 126: npe(()->b1.setProperty("sectionsonly", null)); > 127: iae(()->b1.setProperty("sectionsonly", "OK")); > 128: npe(()->b1.setProperty("sectionsonly", null)); Is 'altsigner' support removed? But I saw it being used in later part of this test. The javadoc for JarSigner.Builder only lists a subset of above properties. Are those not in javadoc internal and can be removed any time, just curious? Nit: maybe add 8242068 to `@bug line` for existing regression tests? ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 01:53:14 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 01:53:14 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: On Wed, 14 Oct 2020 00:00:38 GMT, Valerie Peng wrote: >> Weijun Wang has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. > > src/java.base/share/classes/sun/security/util/SignatureUtil.java line 164: > >> 162: } >> 163: } else { >> 164: paramSpec = getDefaultAlgorithmParameterSpec(sigName, null); > > Same comment here as for the getParamSpec(String, AlgorithmParameters) above. Given that there are two methods named > getParamSpec(...), maybe add a boolean argument to indicate whether to return default value if the to-be-converted > object is null? Same reply. I'll see if it's OK to return a null. > src/java.base/share/classes/sun/security/util/SignatureUtil.java line 290: > >> 288: * that must be initialized with a AlgorithmParameterSpec now. >> 289: */ >> 290: public static AlgorithmParameterSpec getDefaultAlgorithmParameterSpec( > > Maybe we can just use "getDefaultParamSpec" to match other methods' name in this class. Just a suggestion. Sure. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 02:04:10 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 02:04:10 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Wed, 14 Oct 2020 04:03:46 GMT, Valerie Peng wrote: >> Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: >> >> signing time, jarsigner -directsign, and digest algorithm check > > src/java.base/share/classes/sun/security/util/KnownOIDs.java line 147: > >> 145: SHAKE128("2.16.840.1.101.3.4.2.11"), >> 146: SHAKE256("2.16.840.1.101.3.4.2.12"), >> 147: SHAKE256_LEN("2.16.840.1.101.3.4.2.18", "SHAKE256-LEN"), > > Can we move this down a little? The ordering within the section is based on the oid value. It's easier to check for > unlisted/unsupported oid value this way. Why not also add SHAKE128_LEN? I'll add one. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 02:18:13 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 02:18:13 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Wed, 14 Oct 2020 05:31:33 GMT, Valerie Peng wrote: >> Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: >> >> signing time, jarsigner -directsign, and digest algorithm check > > src/java.base/share/classes/sun/security/util/SignatureUtil.java line 218: > >> 216: * >> 217: * @param signer Signature object that tells you RSASA-PSS params >> 218: * @param sigalg Signature algorithm tells you who with who > > who with who? Maybe I was thinking about SHA1withRSA, but here it can be the new algorithms. Just remove the confusing words. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From valeriep at openjdk.java.net Fri Oct 16 02:18:14 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 16 Oct 2020 02:18:14 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Wed, 14 Oct 2020 03:51:23 GMT, Weijun Wang wrote: >> Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: >> >> - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner >> >> - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature >> algorithms >> >> - A new JarSigner property "directsign" >> >> - Updating the jarsigner tool doc >> >> Major code changes: >> >> - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm >> there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. >> >> - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java >> >> - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId >> >> - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing >> >> - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms >> >> - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed > > Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: > > signing time, jarsigner -directsign, and digest algorithm check test/lib/jdk/test/lib/security/timestamp/TsaSigner.java line 221: > 219: new X500Name(issuerName), > 220: signerEntry.cert.getSerialNumber(), > 221: AlgorithmId.get(SignatureUtil.extractDigestAlgFromDwithE(sigAlgo)), So, sigAlgo would never be RSASSA-PSS, EDDSA, ED25519, or ED448? ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 02:23:17 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 02:23:17 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Wed, 14 Oct 2020 19:18:04 GMT, Valerie Peng wrote: >> Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: >> >> signing time, jarsigner -directsign, and digest algorithm check > > src/java.base/share/classes/sun/security/pkcs/PKCS7.java line 172: > >> 170: throws IOException >> 171: { >> 172: ContentInfo block = new ContentInfo(derin, oldStyle); > > With this change, i.e. using a local variable instead of setting the field 'contentInfo', the 'contentInfo' field seems > to left unset when contentType equals to ContentInfo.NETSCAPE_CERT_SEQUENCE_OID? I'll see what the best code is, but I don't like the way contentInfo is assigned twice, once as the whole block and once as the content inside. I'd rather add a `contentInfo = block` in its else if block. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 02:34:19 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 02:34:19 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Thu, 15 Oct 2020 02:03:13 GMT, Valerie Peng wrote: >> Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: >> >> signing time, jarsigner -directsign, and digest algorithm check > > src/java.base/share/classes/sun/security/pkcs/PKCS7.java line 818: > >> 816: DerOutputStream derSigAlg = new DerOutputStream(); >> 817: sigAlgID.derEncode(derSigAlg); >> 818: derAlgs.writeImplicit((byte)0xA1, derSigAlg); > > Are you sure that this context specific tag value is implicit? In RFC 6211, some other ASN.1 definition uses IMPLICIT > keyword after the [x] which seems to suggest that the default is explicit unless specified. Besides, the layman's guide > sec2.3 also states "The keyword [class number] alone is the same as explicit tagging, except when the "module" in which > the ASN.1 type is defined has implicit tagging by default." So, it seems that explicit tagging should be the default? In the formal definition at https://tools.ietf.org/html/rfc6211#appendix-A, you can see `DEFINITIONS IMPLICIT TAGS` covers from BEGIN to END. Those explicit IMPLICIT tags you see are CMS ASN.1 definitions, and it looks in its own RFC at https://tools.ietf.org/html/rfc5652#section-12, IMPLICIT and EXPLICIT are always written out. I can confirm both OpenSSL and BC use IMPLICIT. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 02:37:13 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 02:37:13 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: <1l-DR1mXVPpbFJQreBp3d7Wt4piE97srR04HUIdth1E=.617a854c-692a-4609-baec-071ee6724d4e@github.com> On Thu, 15 Oct 2020 20:42:30 GMT, Valerie Peng wrote: >> Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: >> >> signing time, jarsigner -directsign, and digest algorithm check > > src/java.base/share/classes/sun/security/pkcs/SignerInfo.java line 549: > >> 547: return encAlg; >> 548: default: >> 549: String digAlg = digAlgId.getName().replace("-", ""); > > This may be incorrect if the digest algorithm is in the SHA3 family. Maybe we should check and apply this conversion > only when digest algorithm starts with "SHA-". Good suggestion. I'll also try some tests. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 02:43:13 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 02:43:13 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: <4gXfJ2y2xbhy9qmP5UOXuPBJA8QC6jyOkKjOVfbj-t8=.f02769e7-e513-4290-9c0f-32d603255742@github.com> On Fri, 16 Oct 2020 01:40:48 GMT, Valerie Peng wrote: >> Weijun Wang has refreshed the contents of this pull request, and previous commits have been removed. The incremental >> views will show differences compared to the previous content of the PR. > > test/jdk/jdk/security/jarsigner/Spec.java line 128: > >> 126: npe(()->b1.setProperty("sectionsonly", null)); >> 127: iae(()->b1.setProperty("sectionsonly", "OK")); >> 128: npe(()->b1.setProperty("sectionsonly", null)); > > Is 'altsigner' support removed? But I saw it being used in later part of this test. The javadoc for JarSigner.Builder > only lists a subset of above properties. Are those not in javadoc internal and can be removed any time, just curious? > Nit: maybe add 8242068 to `@bug line` for existing regression tests? It's deprecated. I remember we are already thinking about deprecating the altSign mechanism when creating the JarSigner API and therefore we didn't add them to the spec from the beginning. Bug id added. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 02:47:11 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 02:47:11 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Fri, 16 Oct 2020 02:15:08 GMT, Valerie Peng wrote: >> Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: >> >> signing time, jarsigner -directsign, and digest algorithm check > > test/lib/jdk/test/lib/security/timestamp/TsaSigner.java line 221: > >> 219: new X500Name(issuerName), >> 220: signerEntry.cert.getSerialNumber(), >> 221: AlgorithmId.get(SignatureUtil.extractDigestAlgFromDwithE(sigAlgo)), > > So, sigAlgo would never be RSASSA-PSS, EDDSA, ED25519, or ED448? There were no getDigestAlgInPkcs7SignerInfo in my initial version. I'll use this new method instead. I'll probably need to try if the new algorithms really work here. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From weijun at openjdk.java.net Fri Oct 16 04:14:11 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 04:14:11 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: <1l-DR1mXVPpbFJQreBp3d7Wt4piE97srR04HUIdth1E=.617a854c-692a-4609-baec-071ee6724d4e@github.com> References: <1l-DR1mXVPpbFJQreBp3d7Wt4piE97srR04HUIdth1E=.617a854c-692a-4609-baec-071ee6724d4e@github.com> Message-ID: On Fri, 16 Oct 2020 02:34:35 GMT, Weijun Wang wrote: >> src/java.base/share/classes/sun/security/pkcs/SignerInfo.java line 549: >> >>> 547: return encAlg; >>> 548: default: >>> 549: String digAlg = digAlgId.getName().replace("-", ""); >> >> This may be incorrect if the digest algorithm is in the SHA3 family. Maybe we should check and apply this conversion >> only when digest algorithm starts with "SHA-". > > Good suggestion. I'll also try some tests. In fact, since now I directly write the signature algorithm into the `SignerInfo.digestEncryptionAlgorithmId` field, the code above is not used at all. The `makeSigAlg` method directly returns the `encAlgId` argument if it has "with" inside. I'll fix it anyway. I've confirmed that if I still write only the key algorithm there (Ex: "EC") then the verification process will see a problem without your suggested change. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From jvernee at openjdk.java.net Fri Oct 16 10:01:11 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Fri, 16 Oct 2020 10:01:11 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> Message-ID: On Thu, 15 Oct 2020 22:39:50 GMT, Coleen Phillimore wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Re-add file erroneously deleted (detected as rename) > > src/hotspot/cpu/x86/foreign_globals_x86.cpp line 2: > >> 1: /* >> 2: * Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. > > Copyright should be 2020. All the new files should have 2020 as the copyright, a bunch don't. Ok, will go and check them. FWIW, this file was added back in 2018 in the panama repo. But, I suppose it is considered new here? ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Fri Oct 16 10:54:21 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Fri, 16 Oct 2020 10:54:21 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v12] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 15 commits: - Back-port of TestByteBuffer fix - Merge branch 'master' into 8254162 - Merge branch 'master' into 8254162 - Merge branch 'master' into 8254162 - Remove spurious check on MemoryScope::confineTo Added tests to make sure no spurious exception is thrown when: * handing off a segment from A to A * sharing an already shared segment - Merge branch 'master' into 8254162 - Simplify example in the toplevel javadoc - Tweak support for mapped memory segments - Tweak referenced to MemoryAddressProxy in Utils.java - Fix performance issue with "small" segment mismatch - ... and 5 more: https://git.openjdk.java.net/jdk/compare/1742c44a...6091ed0f ------------- Changes: https://git.openjdk.java.net/jdk/pull/548/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=11 Stats: 8110 lines in 79 files changed: 5403 ins; 1530 del; 1177 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From jvernee at openjdk.java.net Fri Oct 16 10:57:12 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Fri, 16 Oct 2020 10:57:12 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> Message-ID: <_goqdHgVLP0vMwrSHTTdfsHui0Etdcl5rBGB_8ksII8=.8f2fdc3f-7f67-4434-9096-69b9a64b50d9@github.com> On Thu, 15 Oct 2020 22:44:54 GMT, Coleen Phillimore wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Re-add file erroneously deleted (detected as rename) > > src/hotspot/cpu/x86/foreign_globals_x86.hpp line 32: > >> 30: #define __ _masm-> >> 31: >> 32: struct VectorRegister { > > Why are these structs and not classes? The fields are meant to be accessed directly, so I went with `struct` since the members default to public. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From jvernee at openjdk.java.net Fri Oct 16 11:01:14 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Fri, 16 Oct 2020 11:01:14 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> Message-ID: On Thu, 15 Oct 2020 22:52:14 GMT, Coleen Phillimore wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Re-add file erroneously deleted (detected as rename) > > src/hotspot/cpu/x86/sharedRuntime_x86_64.cpp line 3885: > >> 3883: >> 3884: __ flush(); >> 3885: } > > I think as a future RFE we should refactor this function and generate_native_wrapper since they're similar (this is > nicer to read). If I can remove is_critical_native code they will be more similar. Yes, I've had similar thoughts as well. This is meant to be temporary code any ways. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From jvernee at openjdk.java.net Fri Oct 16 11:15:11 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Fri, 16 Oct 2020 11:15:11 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> Message-ID: On Thu, 15 Oct 2020 22:42:49 GMT, Coleen Phillimore wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Re-add file erroneously deleted (detected as rename) > > src/hotspot/cpu/x86/foreign_globals_x86.cpp line 56: > >> 54: } >> 55: >> 56: const ABIDescriptor parseABIDescriptor(JNIEnv* env, jobject jabi) { > > I don't know if you care about performance but of these env->calls transition into the VM and back out again. You > should prefix all the code that comes from java to native with JNI_ENTRY and just use native JVM code to implement > these. Currently this is prefixed with `JVM_ENTRY` e.g. like: JVM_ENTRY(jlong, PI_generateAdapter(JNIEnv* env, jclass _unused, jobject abi, jobject layout)) { ThreadToNativeFromVM ttnfvm(thread); return ProgrammableInvoker::generate_adapter(env, abi, layout); } JVM_END (where `generate_adapter` ends up calling `parseABIDescriptor`) JVM_ENTYRY seems to be mostly the same except for JNI_ENTRY having a `WeakPreserverExceptionMark` as well. Do we need to switch these? Also, I guess if we want to use VM code directly, we should get rid of the `ThreadToNativeFromVM` RAII handle. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From weijun at openjdk.java.net Fri Oct 16 15:51:25 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 16 Oct 2020 15:51:25 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v8] In-Reply-To: References: Message-ID: <36jLvwvRrwAOu4jRH4fSHmhvW0h7jLfgD2x8ia4TJ34=.9acb793c-1fcb-42d0-91e7-14e597a63003@github.com> > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: no default getParamSpec, renames, comments, more testing ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/322/files - new: https://git.openjdk.java.net/jdk/pull/322/files/734fd034..bd3a1596 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=07 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=06-07 Stats: 101 lines in 9 files changed: 60 ins; 12 del; 29 mod Patch: https://git.openjdk.java.net/jdk/pull/322.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/322/head:pull/322 PR: https://git.openjdk.java.net/jdk/pull/322 From valeriep at openjdk.java.net Fri Oct 16 19:07:12 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 16 Oct 2020 19:07:12 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v7] In-Reply-To: References: Message-ID: On Fri, 16 Oct 2020 02:30:55 GMT, Weijun Wang wrote: >> src/java.base/share/classes/sun/security/pkcs/PKCS7.java line 818: >> >>> 816: DerOutputStream derSigAlg = new DerOutputStream(); >>> 817: sigAlgID.derEncode(derSigAlg); >>> 818: derAlgs.writeImplicit((byte)0xA1, derSigAlg); >> >> Are you sure that this context specific tag value is implicit? In RFC 6211, some other ASN.1 definition uses IMPLICIT >> keyword after the [x] which seems to suggest that the default is explicit unless specified. Besides, the layman's guide >> sec2.3 also states "The keyword [class number] alone is the same as explicit tagging, except when the "module" in which >> the ASN.1 type is defined has implicit tagging by default." So, it seems that explicit tagging should be the default? > > In the formal definition at https://tools.ietf.org/html/rfc6211#appendix-A, you can see `DEFINITIONS IMPLICIT TAGS` > covers from BEGIN to END. Those explicit IMPLICIT tags you see are CMS ASN.1 definitions, and it looks in its own RFC > at https://tools.ietf.org/html/rfc5652#section-12, IMPLICIT and EXPLICIT are always written out. I can confirm both > OpenSSL and BC use IMPLICIT. Ah, I see. There is a line about implicit tags as you pointed out. Good~ >> src/java.base/share/classes/sun/security/pkcs/PKCS7.java line 172: >> >>> 170: throws IOException >>> 171: { >>> 172: ContentInfo block = new ContentInfo(derin, oldStyle); >> >> With this change, i.e. using a local variable instead of setting the field 'contentInfo', the 'contentInfo' field seems >> to left unset when contentType equals to ContentInfo.NETSCAPE_CERT_SEQUENCE_OID? > > I'll see what the best code is, but I don't like the way contentInfo is assigned twice, once as the whole block and > once as the content inside. I'd rather add a `contentInfo = block` in its else if block. Right, I also dislike the double assignment. Just making sure that contentInfo is set somewhere. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From valeriep at openjdk.java.net Fri Oct 16 19:36:13 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 16 Oct 2020 19:36:13 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v6] In-Reply-To: References: Message-ID: On Fri, 16 Oct 2020 01:38:44 GMT, Weijun Wang wrote: >> src/java.base/share/classes/sun/security/util/SignatureUtil.java line 94: >> >>> 92: * @return an AlgorithmParameterSpec object >>> 93: * @throws ProviderException >>> 94: */ >> >> Well, I am a bit unsure about your changes to this method. With your change, it returns default parameter spec (instead >> of null) when the specified AlgorithmParameters object is null. This may not be desirable for all cases? Existing >> callers would have to check for (params != null) before calling this method. The javadoc description also seems a bit >> strange with the to-be-converted AlgorithmParameters object being optional. Maybe add a separate method like >> `getParamSpecWithDefault` on top of this method or add a separate boolean argument `useDefault`? > > I cannot remember why I need to return a default. The only default we current have is for RSASSA-PSS, and in all > RSASSA-PSS AlgorithmId for signature I see there is always the params. (When it's for a key the params can be missing). > All 3 callers of this method is on a signature AlgorithmId so the params should not be null. I'll remove the default > return value and do more testing. Sounds good. RSASSA-PSS sig algorithm id always have params, it's required. ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From valeriep at openjdk.java.net Fri Oct 16 20:04:16 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 16 Oct 2020 20:04:16 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v8] In-Reply-To: <36jLvwvRrwAOu4jRH4fSHmhvW0h7jLfgD2x8ia4TJ34=.9acb793c-1fcb-42d0-91e7-14e597a63003@github.com> References: <36jLvwvRrwAOu4jRH4fSHmhvW0h7jLfgD2x8ia4TJ34=.9acb793c-1fcb-42d0-91e7-14e597a63003@github.com> Message-ID: On Fri, 16 Oct 2020 15:51:25 GMT, Weijun Wang wrote: >> Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: >> >> - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner >> >> - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature >> algorithms >> >> - A new JarSigner property "directsign" >> >> - Updating the jarsigner tool doc >> >> Major code changes: >> >> - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm >> there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. >> >> - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java >> >> - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId >> >> - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing >> >> - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms >> >> - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed > > Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: > > no default getParamSpec, renames, comments, more testing Marked as reviewed by valeriep (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From david.holmes at oracle.com Sun Oct 18 22:09:06 2020 From: david.holmes at oracle.com (David Holmes) Date: Mon, 19 Oct 2020 08:09:06 +1000 Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> Message-ID: Hi Jorn, I'm not reviewing this but this exchange caught my attention ... On 16/10/2020 9:15 pm, Jorn Vernee wrote: > On Thu, 15 Oct 2020 22:42:49 GMT, Coleen Phillimore wrote: > >>> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >>> >>> Re-add file erroneously deleted (detected as rename) >> >> src/hotspot/cpu/x86/foreign_globals_x86.cpp line 56: >> >>> 54: } >>> 55: >>> 56: const ABIDescriptor parseABIDescriptor(JNIEnv* env, jobject jabi) { >> >> I don't know if you care about performance but of these env->calls transition into the VM and back out again. You >> should prefix all the code that comes from java to native with JNI_ENTRY and just use native JVM code to implement >> these. > > Currently this is prefixed with `JVM_ENTRY` e.g. like: > JVM_ENTRY(jlong, PI_generateAdapter(JNIEnv* env, jclass _unused, jobject abi, jobject layout)) > { > ThreadToNativeFromVM ttnfvm(thread); > return ProgrammableInvoker::generate_adapter(env, abi, layout); > } > JVM_END > (where `generate_adapter` ends up calling `parseABIDescriptor`) > > JVM_ENTYRY seems to be mostly the same except for JNI_ENTRY having a `WeakPreserverExceptionMark` as well. Do we need > to switch these? Also, I guess if we want to use VM code directly, we should get rid of the `ThreadToNativeFromVM` RAII > handle. Why are you going from native to VM to native again with this code? You would use a JNI/JVM_ENTRY because you have to execute VM runtime code. But your code immediately switches back to native and doesn't execute any VM runtime code (other than that involved in the transition logic itself). ?? Cheers, David > ------------- > > PR: https://git.openjdk.java.net/jdk/pull/634 > From mcimadamore at openjdk.java.net Mon Oct 19 10:34:32 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Mon, 19 Oct 2020 10:34:32 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v13] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation > (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from > multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee > that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class > has been added, which defines several useful dereference routines; these are really just thin wrappers around memory > access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not > the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link > to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit > of dereference. This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which > wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as > dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability > in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; > secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can > use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done > by calling `MemoryAddress::asSegmentRestricted`). A list of the API, implementation and test changes is provided > below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be > happy to point at existing discussions, and/or to provide the feedback required. A big thank to Erik Osterlund, > Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd > like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. Thanks Maurizio > Javadoc: http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative > to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a > carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access > base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte > offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. > `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which > it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both > `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients > can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory > access var handle support. > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to > achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment > is shared, it would be possible for a thread to close it while another is accessing it. After considering several > options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he > reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world > (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a > close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and > the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). > It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. Sadly, none of > these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, > we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to > whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of > stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). The question is, then, > once we detect that a thread is accessing the very segment we're about to close, what should happen? We first > experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it > fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the > machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to > minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread > is accessing the segment being closed. As written in the javadoc, this doesn't mean that clients should just catch and > try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should > be treated as such. In terms of gritty implementation, we needed to centralize memory access routines in a single > place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in > addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` > annotation, which tells the VM that something important is going on. To achieve this, we created a new (autogenerated) > class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, > like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is > tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during > access (which is important when registering segments against cleaners). Of course, to make memory access safe, memory > access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead > of unsafe, so that a liveness check can be triggered (in case a scope is present). `ScopedMemoryAccess` has a > `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed > successfully. The implementation of `MemoryScope` (now significantly simplified from what we had before), has two > implementations, one for confined segments and one for shared segments; the main difference between the two is what > happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared > segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or > `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` > state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. #### > Memory access var handles overhaul The key realization here was that if all memory access var handles took a > coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle > form. This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var > handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that > e.g. additional offset is injected into a base memory access var handle. This also helped in simplifying the > implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level > access on the innards of the memory access var handle. All that code is now gone. #### Test changes Not much to see > here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, > since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` > functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the > microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared > segment case. [1] - https://openjdk.java.net/jeps/393 [2] - https://openjdk.java.net/jeps/389 [3] - > https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Address CSR comments ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/6091ed0f..31674311 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=12 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=11-12 Stats: 1224 lines in 2 files changed: 135 ins; 737 del; 352 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Mon Oct 19 10:39:14 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Mon, 19 Oct 2020 10:39:14 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v6] In-Reply-To: References: Message-ID: On Mon, 12 Oct 2020 18:06:55 GMT, Maurizio Cimadamore wrote: >> Build changes look good. > > I've just uploaded a biggie update to the foreign memory access support. While doing performance evaluation, we have > realized that mixing a multi-level hierarchy (`MappedMemorySegment extends MemorySegments`) with exact invoke semantics > of `VarHandle` and `MethodHandle` is not a good match and can lead to great performance degradation for seemingly > "correct" code. While some of this can be attributed to the `VarHandle` API, or to the fact that the so called > "generic" invocation path should not be that slow in case where the parameters are clearly related, it seems smelly > that a primitive API such as `MemorySegment` should give raise to such issues. We have therefore decided to drop the > `MappedMemorySegment` - this means that there's only one memory segment type users can deal with: `MemorySegment` - and > no chance for mistakes. Of course `MappedMemorySegment` has been primarily introduces to allow for operations which > were previously possible on `MappedByteBuffer` such as `force`. To support these use cases, a separate class has been > introduced, namely `MappedMemorySegments` (note the trailing `S`). This class contains a bunch of static methods which > can be used to achieve the desired effects, without polluting the `MemorySegment` API. A new method has been added on > `MemorySegment` which returns an optional file descriptor; this might be useful for clients which want to guess whether > a segment is in fact a mapped segment, or if they need (e.g. in Windows) the file descriptor to do some other kind of > low level op. I think this approach is more true to the goals and spirit of the Foreign Memory Access API, and it also > offers some ways to improve over the existing API: for instance, the only reason why the `MemorySegment::spliterator` > method was a static method was that we needed inference, so that we could return either a `Spliterator` > or a `Spliterator`. All of that is gone now, so the method can return to be what it morally always > has been: an instance method on `MemorySegment`. Updated javadoc: > http://cr.openjdk.java.net/~mcimadamore/8254162_v2/javadoc/jdk/incubator/foreign/package-summary.html Updated > specdiff: http://cr.openjdk.java.net/~mcimadamore/8254162_v2/specdiff/overview-summary.html The latest iteration addresses a comment raised during CSR review; more specifically, the `MemoryAccess` class has several variants of dereference methods - e.g. `getInt`, `getInt_LE`, `getInt_BE`, to support different endianness. The comment was to just have two overloads, e.g. `getInt` and `getInt(ByteOrder)` instead of three. I've implemented the suggestion in this new iteration, as I think it makes the API a bit more compact. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From fyang at openjdk.java.net Mon Oct 19 11:14:22 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Mon, 19 Oct 2020 11:14:22 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v9] In-Reply-To: References: Message-ID: > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. > Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 > > Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. > For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. > "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. > > We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. > Patch passed jtreg tier1-3 tests with QEMU system emulator. > Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make > sure that there's no regression. > We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java > We measured the performance benefit with an aarch64 cycle-accurate simulator. > Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. > > For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on > real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is > detected. Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 11 commits: - Merge master - Remove unnecessary code changes in vm_version_aarch64.cpp - Merge master - Merge master - Merge master - Merge master - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check - Rebase - Merge master - Fix trailing whitespace issue - ... and 1 more: https://git.openjdk.java.net/jdk/compare/e9be2db7...05551701 ------------- Changes: https://git.openjdk.java.net/jdk/pull/207/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=207&range=08 Stats: 1262 lines in 36 files changed: 1007 ins; 22 del; 233 mod Patch: https://git.openjdk.java.net/jdk/pull/207.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/207/head:pull/207 PR: https://git.openjdk.java.net/jdk/pull/207 From jvernee at openjdk.java.net Mon Oct 19 11:29:23 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Mon, 19 Oct 2020 11:29:23 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> Message-ID: On Fri, 16 Oct 2020 11:12:01 GMT, Jorn Vernee wrote: >> src/hotspot/cpu/x86/foreign_globals_x86.cpp line 56: >> >>> 54: } >>> 55: >>> 56: const ABIDescriptor parseABIDescriptor(JNIEnv* env, jobject jabi) { >> >> I don't know if you care about performance but of these env->calls transition into the VM and back out again. You >> should prefix all the code that comes from java to native with JNI_ENTRY and just use native JVM code to implement >> these. > > Currently this is prefixed with `JVM_ENTRY` e.g. like: > JVM_ENTRY(jlong, PI_generateAdapter(JNIEnv* env, jclass _unused, jobject abi, jobject layout)) > { > ThreadToNativeFromVM ttnfvm(thread); > return ProgrammableInvoker::generate_adapter(env, abi, layout); > } > JVM_END > (where `generate_adapter` ends up calling `parseABIDescriptor`) > > JVM_ENTYRY seems to be mostly the same except for JNI_ENTRY having a `WeakPreserverExceptionMark` as well. Do we need > to switch these? Also, I guess if we want to use VM code directly, we should get rid of the `ThreadToNativeFromVM` RAII > handle. re-wrote this code to use the VM internal APIs instead of JNI, changes are isolated in a sub-pr here: https://github.com/mcimadamore/jdk/pull/1 Could you take a look? ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From jvernee at openjdk.java.net Mon Oct 19 11:29:22 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Mon, 19 Oct 2020 11:29:22 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> Message-ID: <1qSzjGTeTsGkvOvOIiXjY8JP944k3uLaq6KhkUP1vHE=.068f0409-ef1d-486a-8d74-c587619893e9@github.com> On Thu, 15 Oct 2020 23:15:07 GMT, Coleen Phillimore wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Re-add file erroneously deleted (detected as rename) > > I looked through some Hotspot runtime code and that looks ok. I saw a couple of strange things on my way through the > code. See comments. Hi David, this code somewhat predates me, so I initially kept the JVM_ENTRY since that was what was already in place. IIRC the thread state transition was added later to be able to call JNI code, which checks that the thread state is native in some asserts. I've re-written this code, per @coleenp 's suggestion, to use VM code directly to replace what we were doing with JNI, so the thread state transition is also gone. I've looked at some of the *_ENTRY macros and the only one that seems to avoid the thread state transition is JVM_LEAF. I've switched the RegisterNatives functions we use to JVM_LEAF to avoid the redundant transitions. I also tried changing `PI_invokeNative` to JVM_LEAF, but since we can call back into Java from that, I run into a missing handle mark assert for some of the tests, so I've left that one as JVM_ENTRY (but removed some redundant braces). I've created a separate sub-pr against this PR's branch to make it easier to see what I've changed: https://github.com/mcimadamore/jdk/pull/1 (feel free to take a look). Thanks for the comments. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Mon Oct 19 13:13:27 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Mon, 19 Oct 2020 13:13:27 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v5] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and > associated pull request [3]). > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate > JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the > writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be > used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, > I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be > periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as > possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you > see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to > Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library > by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory > layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native > function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes > a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a > `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, > and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which > acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures > (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout > attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take > place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and > back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than > allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to > use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a > performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing > native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For > instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, > in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is > a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as > it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes > associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library > loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to > JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes > are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing > some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request > into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native > scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are > implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support > the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of > some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale > behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a > Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function > (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding > native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward > fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on > Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that > kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no > longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a > floating point value, or an integral value; this knowledge is required because floating points are passed in different > registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which > contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this > attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive > operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such > bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the > main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` > what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the > `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the > various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed > below: > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see > `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating > a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The > buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in > their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This > is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is > some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, > we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above > (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer > allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of > bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), > then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an > intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to > add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond > what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more > readings on the internals of the foreign linker support, please refer to [5]. > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) > which aim at testing the linker from the perspective of code that clients could write. But we also have deeper > combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI > implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s > for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the > linker machinery as a black box and verify that the support works by checking that the native call returned the results > we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also > mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing > on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - > https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - > https://git.openjdk.java.net/jdk/pull/548 [4] - > https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - > http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Use separate constants for native invoker code size ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/634/files - new: https://git.openjdk.java.net/jdk/pull/634/files/830c5cea..c595a8dd Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=03-04 Stats: 12 lines in 4 files changed: 8 ins; 0 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Mon Oct 19 15:01:29 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Mon, 19 Oct 2020 15:01:29 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v6] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and > associated pull request [3]). > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate > JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the > writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be > used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, > I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be > periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as > possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you > see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to > Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library > by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory > layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native > function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes > a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a > `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, > and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which > acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures > (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout > attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take > place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and > back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than > allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to > use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a > performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing > native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For > instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, > in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is > a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as > it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes > associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library > loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to > JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes > are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing > some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request > into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native > scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are > implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support > the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of > some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale > behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a > Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function > (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding > native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward > fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on > Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that > kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no > longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a > floating point value, or an integral value; this knowledge is required because floating points are passed in different > registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which > contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this > attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive > operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such > bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the > main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` > what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the > `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the > various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed > below: > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see > `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating > a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The > buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in > their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This > is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is > some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, > we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above > (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer > allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of > bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), > then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an > intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to > add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond > what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more > readings on the internals of the foreign linker support, please refer to [5]. > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) > which aim at testing the linker from the perspective of code that clients could write. But we also have deeper > combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI > implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s > for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the > linker machinery as a black box and verify that the support works by checking that the native call returned the results > we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also > mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing > on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - > https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - > https://git.openjdk.java.net/jdk/pull/548 [4] - > https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - > http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request incrementally with two additional commits since the last revision: - Fix incorrect capitalization in one copyright header - Update copyright years, and add classpath exception to files that were missing it ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/634/files - new: https://git.openjdk.java.net/jdk/pull/634/files/c595a8dd..7d6eadc7 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=04-05 Stats: 117 lines in 56 files changed: 56 ins; 0 del; 61 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From kvn at openjdk.java.net Mon Oct 19 20:29:18 2020 From: kvn at openjdk.java.net (Vladimir Kozlov) Date: Mon, 19 Oct 2020 20:29:18 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v9] In-Reply-To: References: Message-ID: On Mon, 19 Oct 2020 11:14:22 GMT, Fei Yang wrote: >> Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com >> >> This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. >> Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 >> >> Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. >> For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. >> "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. >> >> We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. >> Patch passed jtreg tier1-3 tests with QEMU system emulator. >> Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make >> sure that there's no regression. >> We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java >> We measured the performance benefit with an aarch64 cycle-accurate simulator. >> Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. >> >> For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on >> real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is >> detected. > > Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains > 11 commits: > - Merge master > - Remove unnecessary code changes in vm_version_aarch64.cpp > - Merge master > - Merge master > - Merge master > - Merge master > - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check > - Rebase > - Merge master > - Fix trailing whitespace issue > - ... and 1 more: https://git.openjdk.java.net/jdk/compare/e9be2db7...05551701 Always run graalunit testing with new intrinsics. You need to adjust Graal test: src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java ------------- Changes requested by kvn (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/207 From weijun at openjdk.java.net Tue Oct 20 02:22:27 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 20 Oct 2020 02:22:27 GMT Subject: RFR: 8242068: Signed JAR support for RSASSA-PSS and EdDSA [v9] In-Reply-To: References: Message-ID: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature > algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm > there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: tiny help screen change ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/322/files - new: https://git.openjdk.java.net/jdk/pull/322/files/bd3a1596..251373f4 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=08 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=322&range=07-08 Stats: 3 lines in 2 files changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/322.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/322/head:pull/322 PR: https://git.openjdk.java.net/jdk/pull/322 From fyang at openjdk.java.net Tue Oct 20 08:07:24 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Tue, 20 Oct 2020 08:07:24 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v9] In-Reply-To: References: Message-ID: On Mon, 19 Oct 2020 20:26:22 GMT, Vladimir Kozlov wrote: > Always run graalunit testing with new intrinsics. You need to adjust Graal test: > src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java Thanks for looking at this. We did run graalunit testing and added the following change in our first commit: diff --git a/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java b/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java index f0e17947460..8f3f4ed9323 100644 --- a/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java +++ b/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java @@ -601,6 +601,7 @@ public class CheckGraalIntrinsics extends GraalTest { if (!config.useSHA512Intrinsics()) { add(ignore, "sun/security/provider/SHA5." + shaCompressName + "([BI)V"); } + add(toBeInvestigated, "sun/security/provider/SHA3." + shaCompressName + "([BI)V"); } ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From fyang at openjdk.java.net Tue Oct 20 13:42:27 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Tue, 20 Oct 2020 13:42:27 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v10] In-Reply-To: References: Message-ID: > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. > Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 > > Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. > For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. > "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. > > We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. > Patch passed jtreg tier1-3 tests with QEMU system emulator. > Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make > sure that there's no regression. > We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java > We measured the performance benefit with an aarch64 cycle-accurate simulator. > Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. > > For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on > real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is > detected. Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 13 commits: - Fix trailing whitespace issue reported by jcheck - Merge master - Merge master - Remove unnecessary code changes in vm_version_aarch64.cpp - Merge master - Merge master - Merge master - Merge master - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check - Rebase - ... and 3 more: https://git.openjdk.java.net/jdk/compare/cdc8c401...d32c8ad7 ------------- Changes: https://git.openjdk.java.net/jdk/pull/207/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=207&range=09 Stats: 1262 lines in 36 files changed: 1007 ins; 22 del; 233 mod Patch: https://git.openjdk.java.net/jdk/pull/207.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/207/head:pull/207 PR: https://git.openjdk.java.net/jdk/pull/207 From silvermangb at gmail.com Tue Oct 20 00:06:39 2020 From: silvermangb at gmail.com (Greg Silverman) Date: Mon, 19 Oct 2020 17:06:39 -0700 Subject: openjdk 11, tomcat for FIPS, and, rhel7 Message-ID: The Red Hat Customer Portal has an article on configuring Tomcat for FIPS mode using NSS for RHEL6. One of the steps is to change the java.security file as follows change the line from security.provider.4=com.sun.net.ssl.internal.ssl.Provider to security.provider.4=com.sun.net.ssl.internal.ssl.Provider SunPKCS11-NSS However, for OpenJDK 11 on RHEL 7 none of the security providers are com.sun.net.ssl.internal.ssl.Provider. What is the correct way to change this for OpenJDK 11? Greg Silverman Principal Software Engineer Veritas Technologies Santa Clara, CA -------------- next part -------------- An HTML attachment was scrubbed... URL: From etlixiang at 126.com Tue Oct 20 08:33:03 2020 From: etlixiang at 126.com (=?GBK?B?y67D28zStcTM7L/V?=) Date: Tue, 20 Oct 2020 16:33:03 +0800 (CST) Subject: how to reproduct JDK-8067648 in jdk8u45 Message-ID: <449af686.4abf.1754523ea5e.Coremail.etlixiang@126.com> Dear expert: When we used jdk8u45 in our system,It had happen a bug that "J 24767 C2 com.sun.crypto.provider.GCTR.doFinal([BII[BI)I (130 bytes) @ 0x00007f0a9687fb61 [0x00007f0a9687f900+0x261]".We knew it is a bug named JDK-8067648 from website?add is "http://openjdk.5641.n7.nabble.com/9-RFR-S-8067648-JVM-crashes-reproducable-with-GCM-cipher-suites-in-GCTR-doFinal-td222764.html",and the email tells us that "8067648: JVM crashes reproducable with GCM cipher suites in GCTR doFinal".But we don't know how to reproduct the bug,Could you teach us the detail step to do it? Thank you and best regards. Xiang Li -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcimadamore at openjdk.java.net Tue Oct 20 17:23:26 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 20 Oct 2020 17:23:26 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and > associated pull request [3]). > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate > JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the > writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be > used by clients. Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, > I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be > periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as > possible. A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you > see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to > Paul Sandoz, who provided many insights (often by trying the bits first hand). Thanks Maurizio > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library > by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory > layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native > function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes > a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a > `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, > and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which > acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures > (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout > attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take > place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and > back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than > allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to > use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a > performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing > native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For > instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, > in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is > a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as > it's the case for other restricted method in the foreign memory API). ### Implementation changes The Java changes > associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library > loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to > JNI library loading (e.g. same library cannot be loaded by different classloaders). As for `NativeScope` the changes > are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing > some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request > into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native > scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are > implemented by two separate subclasses of `AbstractNativeScopeImpl`. Of course the bulk of the changes are to support > the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of > some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale > behind the VM support, with some references to the code [5]. The main idea behind foreign linker is to infer, given a > Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function > (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding > native call targeting the requested native function. This inference scheme can be defined in a pretty straightforward > fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on > Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that > kind of inference. For the inference process to work, we need to attach extra information to memory layouts; it is no > longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a > floating point value, or an integral value; this knowledge is required because floating points are passed in different > registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which > contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this > attribute, and performs classification accordingly. A native call is decomposed into a sequence of basic, primitive > operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such > bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the > main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` > what is the set of bindings associated with the downcall/upcall. At the heart of the foreign linker support is the > `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the > various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed > below: > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see > `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating > a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The > buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in > their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This > is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is > some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, > we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above > (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer > allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of > bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), > then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an > intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to > add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond > what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). Again, for more > readings on the internals of the foreign linker support, please refer to [5]. > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) > which aim at testing the linker from the perspective of code that clients could write. But we also have deeper > combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI > implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s > for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the > linker machinery as a black box and verify that the support works by checking that the native call returned the results > we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also > mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing > on. Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. [1] - > https://openjdk.java.net/jeps/389 [2] - https://openjdk.java.net/jeps/393 [3] - > https://git.openjdk.java.net/jdk/pull/548 [4] - > https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md [5] - > http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 25 commits: - Merge branch 'master' into 8254231_linker - Fix incorrect capitalization in one copyright header - Update copyright years, and add classpath exception to files that were missing it - Use separate constants for native invoker code size - Re-add file erroneously deleted (detected as rename) - Re-add erroneously removed files - Merge branch 'master' into 8254231_linker - Fix tests - Fix more whitespaces - Fix whitespaces - Remove rejected file - ... and 15 more: https://git.openjdk.java.net/jdk/compare/cb6167b2...502bd980 ------------- Changes: https://git.openjdk.java.net/jdk/pull/634/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=06 Stats: 75609 lines in 264 files changed: 72724 ins; 1608 del; 1277 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From xuelei at openjdk.java.net Tue Oct 20 22:24:17 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Tue, 20 Oct 2020 22:24:17 GMT Subject: RFR: 8199697: FIPS 186-4 RSA Key Generation [v2] In-Reply-To: References: Message-ID: On Mon, 12 Oct 2020 22:05:32 GMT, Valerie Peng wrote: >> Could someone please help review this RFE? Update existing RSA key pair generation code following the guidelines from >> FIPS 186-4 and FIPS 186-5 (draft). Current proposed changes updates the prime generation code (for P, Q) based on FIPS >> 186-4 B.3.3 when keysize and public exponent met the requirements set in FIPS 186-4/5. Thanks, >> Valerie > > Valerie Peng has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev > excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since > the last revision: > - Merge branch 'master' into JDK-8199697 > - 8199697: FIPS 186-4 RSA Key Generation > > Changed RSA key pair generation code following the guidelines from FIPS 186-4. test/jdk/sun/security/rsa/SpecTest.java line 33: > 31: * @run main SpecTest 768 > 32: * @run main SpecTest 1024 > 33: * @run main SpecTest 1024 65537 65537 is the default public exponent (see the main() method). So, the two test case is the same: * @run main SpecTest 1024 * @run main SpecTest 1024 65537 Maybe, we can keep the test case for F0, and add a new public exponent number like 167971. src/java.base/share/classes/sun/security/rsa/RSAKeyPairGenerator.java line 200: > 198: if (kp != null) { > 199: return kp; > 200: } The logic may be more clear if moving the checking of n and key generation out of the loop for q, by regenerate both p and q if needed. ------------- PR: https://git.openjdk.java.net/jdk/pull/420 From kvn at openjdk.java.net Tue Oct 20 23:11:19 2020 From: kvn at openjdk.java.net (Vladimir Kozlov) Date: Tue, 20 Oct 2020 23:11:19 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v10] In-Reply-To: References: Message-ID: On Tue, 20 Oct 2020 13:42:27 GMT, Fei Yang wrote: >> Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com >> >> This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. >> Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 >> >> Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. >> For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. >> "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. >> >> We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. >> Patch passed jtreg tier1-3 tests with QEMU system emulator. >> Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make >> sure that there's no regression. >> We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java >> We measured the performance benefit with an aarch64 cycle-accurate simulator. >> Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. >> >> For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on >> real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is >> detected. > > Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains > 13 commits: > - Fix trailing whitespace issue reported by jcheck > - Merge master > - Merge master > - Remove unnecessary code changes in vm_version_aarch64.cpp > - Merge master > - Merge master > - Merge master > - Merge master > - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check > - Rebase > - ... and 3 more: https://git.openjdk.java.net/jdk/compare/cdc8c401...d32c8ad7 Someone in Oracle have to run tier1-tier3 testing with these changes to make sure nothing is broken. I don't want to repeat 8254790. src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java line 604: > 602: add(ignore, "sun/security/provider/SHA5." + shaCompressName + "([BI)V"); > 603: } > 604: add(toBeInvestigated, "sun/security/provider/SHA3." + shaCompressName + "([BI)V"); This should be under `if (isJDK16OrHigher())` check. Something like this: https://github.com/openjdk/jdk/pull/650/files#diff-d1f378fc1b7fe041309e854d40b3a95a91e63fdecf0ecd9826b7c95eaeba314eR527 You can wait when Aleksey push it and update your changes ------------- Changes requested by kvn (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/207 From valeriep at openjdk.java.net Wed Oct 21 00:18:19 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 21 Oct 2020 00:18:19 GMT Subject: RFR: 8199697: FIPS 186-4 RSA Key Generation [v2] In-Reply-To: References: Message-ID: On Tue, 20 Oct 2020 22:21:02 GMT, Xue-Lei Andrew Fan wrote: >> Valerie Peng has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev >> excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since >> the last revision: >> - Merge branch 'master' into JDK-8199697 >> - 8199697: FIPS 186-4 RSA Key Generation >> >> Changed RSA key pair generation code following the guidelines from FIPS 186-4. > > src/java.base/share/classes/sun/security/rsa/RSAKeyPairGenerator.java line 200: > >> 198: if (kp != null) { >> 199: return kp; >> 200: } > > The logic may be more clear if moving the checking of n and key generation out of the loop for q, by regenerate both p > and q if needed. Ok, will add an outer while-loop as in the existing code. ------------- PR: https://git.openjdk.java.net/jdk/pull/420 From valeriep at openjdk.java.net Wed Oct 21 00:23:18 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 21 Oct 2020 00:23:18 GMT Subject: RFR: 8199697: FIPS 186-4 RSA Key Generation [v2] In-Reply-To: References: Message-ID: On Tue, 13 Oct 2020 05:23:01 GMT, Xue-Lei Andrew Fan wrote: >> Valerie Peng has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev >> excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since >> the last revision: >> - Merge branch 'master' into JDK-8199697 >> - 8199697: FIPS 186-4 RSA Key Generation >> >> Changed RSA key pair generation code following the guidelines from FIPS 186-4. > > test/jdk/sun/security/rsa/SpecTest.java line 33: > >> 31: * @run main SpecTest 768 >> 32: * @run main SpecTest 1024 >> 33: * @run main SpecTest 1024 65537 > > 65537 is the default public exponent (see the main() method). So, the two test case is the same: > * @run main SpecTest 1024 > * @run main SpecTest 1024 65537 > > Maybe, we can keep the test case for F0, and add a new public exponent number like 167971. I want to stop using F0 since it's no longer deemed valid for FIPS 186-4. For backward compatibility, we don't reject F0, but perhaps we should stop using it so people will start shifting to use F4. I can replace 3 with 167971. ------------- PR: https://git.openjdk.java.net/jdk/pull/420 From valeriep at openjdk.java.net Wed Oct 21 02:48:35 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 21 Oct 2020 02:48:35 GMT Subject: RFR: 8199697: FIPS 186-4 RSA Key Generation [v3] In-Reply-To: References: Message-ID: > Could someone please help review this RFE? Update existing RSA key pair generation code following the guidelines from > FIPS 186-4 and FIPS 186-5 (draft). Current proposed changes updates the prime generation code (for P, Q) based on FIPS > 186-4 B.3.3 when keysize and public exponent met the requirements set in FIPS 186-4/5. Thanks, > Valerie Valerie Peng has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: - Merge - Updated to address review comments - Merge branch 'master' into JDK-8199697 - 8199697: FIPS 186-4 RSA Key Generation Changed RSA key pair generation code following the guidelines from FIPS 186-4. ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/420/files - new: https://git.openjdk.java.net/jdk/pull/420/files/358fee06..61952af2 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=420&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=420&range=01-02 Stats: 338884 lines in 994 files changed: 321615 ins; 12717 del; 4552 mod Patch: https://git.openjdk.java.net/jdk/pull/420.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/420/head:pull/420 PR: https://git.openjdk.java.net/jdk/pull/420 From xuelei at openjdk.java.net Wed Oct 21 03:21:20 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Wed, 21 Oct 2020 03:21:20 GMT Subject: RFR: 8199697: FIPS 186-4 RSA Key Generation [v3] In-Reply-To: References: Message-ID: On Wed, 21 Oct 2020 02:48:35 GMT, Valerie Peng wrote: >> Could someone please help review this RFE? Update existing RSA key pair generation code following the guidelines from >> FIPS 186-4 and FIPS 186-5 (draft). Current proposed changes updates the prime generation code (for P, Q) based on FIPS >> 186-4 B.3.3 when keysize and public exponent met the requirements set in FIPS 186-4/5. Thanks, >> Valerie > > Valerie Peng has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev > excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since > the last revision: > - Merge > - Updated to address review comments > - Merge branch 'master' into JDK-8199697 > - 8199697: FIPS 186-4 RSA Key Generation > > Changed RSA key pair generation code following the guidelines from FIPS 186-4. Looks good to me. ------------- Marked as reviewed by xuelei (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/420 From fyang at openjdk.java.net Wed Oct 21 09:10:57 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Wed, 21 Oct 2020 09:10:57 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v10] In-Reply-To: References: Message-ID: On Tue, 20 Oct 2020 23:06:41 GMT, Vladimir Kozlov wrote: >> Fei Yang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 13 commits: >> >> - Fix trailing whitespace issue reported by jcheck >> - Merge master >> - Merge master >> - Remove unnecessary code changes in vm_version_aarch64.cpp >> - Merge master >> - Merge master >> - Merge master >> - Merge master >> - Add sha3 instructions to cpu/aarch64/aarch64-asmtest.py and regenerate the test in assembler_aarch64.cpp:asm_check >> - Rebase >> - ... and 3 more: https://git.openjdk.java.net/jdk/compare/cdc8c401...d32c8ad7 > > src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java line 604: > >> 602: add(ignore, "sun/security/provider/SHA5." + shaCompressName + "([BI)V"); >> 603: } >> 604: add(toBeInvestigated, "sun/security/provider/SHA3." + shaCompressName + "([BI)V"); > > This should be under `if (isJDK16OrHigher())` check. Something like this: > https://github.com/openjdk/jdk/pull/650/files#diff-d1f378fc1b7fe041309e854d40b3a95a91e63fdecf0ecd9826b7c95eaeba314eR527 > You can wait when Aleksey push it and update your changes OK. Will update with the following change after Aleksey's PR is integrated: --- a/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java +++ b/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java @@ -608,6 +608,10 @@ public class CheckGraalIntrinsics extends GraalTest { if (!config.useSHA512Intrinsics()) { add(ignore, "sun/security/provider/SHA5." + shaCompressName + "([BI)V"); } + + if (isJDK16OrHigher()) { + add(toBeInvestigated, "sun/security/provider/SHA3." + shaCompressName + "([BI)V"); + } } ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From fyang at openjdk.java.net Wed Oct 21 09:23:57 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Wed, 21 Oct 2020 09:23:57 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v10] In-Reply-To: References: Message-ID: On Tue, 20 Oct 2020 23:08:22 GMT, Vladimir Kozlov wrote: > Someone in Oracle have to run tier1-tier3 testing with these changes to make sure nothing is broken. I don't want to repeat 8254790. That's appreciated. On my side, I run tier1-tier3 both on aarch64 linux and x86_64 linux. The test result on these two platforms looks good for the latest changes. ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From mcimadamore at openjdk.java.net Wed Oct 21 10:44:38 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Wed, 21 Oct 2020 10:44:38 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v8] In-Reply-To: References: Message-ID: <663pAFJdnevrreAcN1Zg9_pkE1kt9Vxxy9unNgQMvDk=.42b1e1ad-192a-418d-9271-c1830dd9fa82@github.com> > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). > > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. > > Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. > > A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). > > Thanks > Maurizio > > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). > > ### Implementation changes > > The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). > > As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. > > Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. > > The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. > > This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. > > For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. > > A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. > > At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: > > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). > > Again, for more readings on the internals of the foreign linker support, please refer to [5]. > > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. > > Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. > > [1] - https://openjdk.java.net/jeps/389 > [2] - https://openjdk.java.net/jeps/393 > [3] - https://git.openjdk.java.net/jdk/pull/548 > [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md > [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Don't use JNI when generating native wrappers ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/634/files - new: https://git.openjdk.java.net/jdk/pull/634/files/502bd980..7cef16f4 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=07 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=06-07 Stats: 478 lines in 17 files changed: 245 ins; 140 del; 93 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Wed Oct 21 11:33:27 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Wed, 21 Oct 2020 11:33:27 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v9] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). > > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. > > Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. > > A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). > > Thanks > Maurizio > > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). > > ### Implementation changes > > The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). > > As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. > > Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. > > The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. > > This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. > > For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. > > A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. > > At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: > > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). > > Again, for more readings on the internals of the foreign linker support, please refer to [5]. > > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. > > Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. > > [1] - https://openjdk.java.net/jeps/389 > [2] - https://openjdk.java.net/jeps/393 > [3] - https://git.openjdk.java.net/jdk/pull/548 > [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md > [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 27 commits: - Merge branch 'master' into 8254231_linker - Don't use JNI when generating native wrappers - Merge branch 'master' into 8254231_linker - Fix incorrect capitalization in one copyright header - Update copyright years, and add classpath exception to files that were missing it - Use separate constants for native invoker code size - Re-add file erroneously deleted (detected as rename) - Re-add erroneously removed files - Merge branch 'master' into 8254231_linker - Fix tests - Fix more whitespaces - ... and 17 more: https://git.openjdk.java.net/jdk/compare/da97ab5c...8c7b75da ------------- Changes: https://git.openjdk.java.net/jdk/pull/634/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=08 Stats: 75713 lines in 267 files changed: 72828 ins; 1608 del; 1277 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From jvernee at openjdk.java.net Wed Oct 21 11:37:17 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Wed, 21 Oct 2020 11:37:17 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v4] In-Reply-To: <1qSzjGTeTsGkvOvOIiXjY8JP944k3uLaq6KhkUP1vHE=.068f0409-ef1d-486a-8d74-c587619893e9@github.com> References: <0Zh0H5gSXzvHSstQ2w8NBM-P8yERRPouvhZJDNGvu4A=.6cde913f-7499-4c45-bc63-b717502b661e@github.com> <2moJ2056gzwWoleYccv21TpFYQHw5h9bA-IZCImplhs=.763198bf-06b0-4589-b01e-217ba84af94a@github.com> <1qSzjGTeTsGkvOvOIiXjY8JP944k3uLaq6KhkUP1vHE=.068f0409-ef1d-486a-8d74-c587619893e9@github.com> Message-ID: On Mon, 19 Oct 2020 11:24:45 GMT, Jorn Vernee wrote: >> I looked through some Hotspot runtime code and that looks ok. I saw a couple of strange things on my way through the code. See comments. > > Hi David, this code somewhat predates me, so I initially kept the JVM_ENTRY since that was what was already in place. IIRC the thread state transition was added later to be able to call JNI code, which checks that the thread state is native in some asserts. > > I've re-written this code, per @coleenp 's suggestion, to use VM code directly to replace what we were doing with JNI, so the thread state transition is also gone. > > I've looked at some of the *_ENTRY macros and the only one that seems to avoid the thread state transition is JVM_LEAF. I've switched the RegisterNatives functions we use to JVM_LEAF to avoid the redundant transitions. I also tried changing `PI_invokeNative` to JVM_LEAF, but since we can call back into Java from that, I run into a missing handle mark assert for some of the tests, so I've left that one as JVM_ENTRY (but removed some redundant braces). > > I've created a separate sub-pr against this PR's branch to make it easier to see what I've changed: https://github.com/mcimadamore/jdk/pull/1 (feel free to take a look). > > Thanks for the comments. I've fixed the following issues from review comments: - don't rely on `MethodHandles::adapter_code_size` (after private discussion) - update copyright years - use VM-internal API instead of JNI when parsing ABIDescriptor and BufferLayout objects while generating down/up call wrappers. As far as I see, that covers all open review comments. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From weijun at openjdk.java.net Wed Oct 21 14:24:22 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Wed, 21 Oct 2020 14:24:22 GMT Subject: Integrated: 8242068: Signed JAR support for RSASSA-PSS and EdDSA In-Reply-To: References: Message-ID: On Wed, 23 Sep 2020 14:41:59 GMT, Weijun Wang wrote: > Major points in CSR at https://bugs.openjdk.java.net/browse/JDK-8245274: > > - new sigalg "RSASSA-PSS", "EdDSA", "Ed25519" and "Ed448" can be used in jarsigner > > - The ".RSA" and ".EC" block extension types (PKCS #7 SignedData inside a signed JAR) are reused for new signature algorithms > > - A new JarSigner property "directsign" > > - Updating the jarsigner tool doc > > Major code changes: > > - Always use the signature algorithm directly as SignerInfo::signatureAlgorithm. We used to use the encryption algorithm there like RSA, DSA, and EC. Now it's always SHA1withRSA or RSASSA-PSS. > > - Move signature related utilities methods from AlgorithmId.java to SignatureUtil.java > > - Add new SignatureUtil methods fromKey() and fromSignature() to simplify creating Signature and getting its AlgorithmId > > - Use the new methods in PKCS10, X509CertImpl, and X509CRLImpl signing > > - Add a new (and intuitive, IMHO) PKCS7::generateNewSignedData capable of all old and new signature algorithms > > - Mark all -altsign related code deprecated and they can be removed once ContentSigner is removed This pull request has now been integrated. Changeset: 839f01dd Author: Weijun Wang URL: https://git.openjdk.java.net/jdk/commit/839f01dd Stats: 1869 lines in 24 files changed: 1132 ins; 558 del; 179 mod 8242068: Signed JAR support for RSASSA-PSS and EdDSA Reviewed-by: valeriep ------------- PR: https://git.openjdk.java.net/jdk/pull/322 From ecki at zusammenkunft.net Wed Oct 21 16:21:09 2020 From: ecki at zusammenkunft.net (Bernd) Date: Wed, 21 Oct 2020 18:21:09 +0200 Subject: ldap.mechsAllowedToSendCredentials - only SASL? Message-ID: Hello, I am looking at 11.0.9 PSU (as of Zulu 11.43-sa) about the CVE-2020-14781 / JDK-8237990 fix and try to understand if my customers might be affected. jdk.jndi.ldap.mechsAllowedToSendCredentials It was not obvious to me, how the mechanism restriction works. According to Oracle and Redhat release notes it only looks at clear / non-TLS. - Can you confirm that SASL with wrapping is not considered as encrypted in this case? - Can you confirm it only applies to SASL based negotiation? (in my test SIMPLE with cleartext passwords works just fine) - Can you confirm it does not apply to "secure" mechanisms like DIGEST-MD5 or different methods like GSSAPI or SIMPLE? Gruss Bernd -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecki at zusammenkunft.net Wed Oct 21 17:26:36 2020 From: ecki at zusammenkunft.net (Bernd) Date: Wed, 21 Oct 2020 19:26:36 +0200 Subject: ldap.mechsAllowedToSendCredentials - only SASL? In-Reply-To: References: Message-ID: BTW: the security patch looks like "simple" is supposed to be rejected when a principal is set, however this is not the case in my tests. Maybe the method is not called correctly in this case? if ("simple".equalsIgnoreCase(authMechanism) && !envprops.containsKey(SECURITY_PRINCIPAL)) { Gruss Bernd Am Mi., 21. Okt. 2020 um 18:21 Uhr schrieb Bernd : > Hello, > > I am looking at 11.0.9 PSU (as of Zulu 11.43-sa) about the CVE-2020-14781 > / JDK-8237990 fix and try to understand if my customers might be > affected. > > jdk.jndi.ldap.mechsAllowedToSendCredentials > > It was not obvious to me, how the mechanism restriction works. > > According to Oracle and Redhat release notes it only looks at clear / > non-TLS. > > - Can you confirm that SASL with wrapping is not considered as encrypted > in this case? > > - Can you confirm it only applies to SASL based negotiation? (in my test > SIMPLE with cleartext passwords works just fine) > > - Can you confirm it does not apply to "secure" mechanisms like DIGEST-MD5 > or different methods like GSSAPI or SIMPLE? > > Gruss > Bernd > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecki at zusammenkunft.net Wed Oct 21 17:30:02 2020 From: ecki at zusammenkunft.net (Bernd) Date: Wed, 21 Oct 2020 19:30:02 +0200 Subject: ldap.mechsAllowedToSendCredentials - only SASL? In-Reply-To: References: Message-ID: And just to add to my confusion, this seems that it only checks when STARTTLS is actually requested but not used? This code really needs revised documentation. + // If current connection is not encrypted, and context seen to be secured with STARTTLS+ // or 'mechsAllowedToSendCredentials' is set to any value via system/context environment properties+ if (!isConnectionEncrypted() && (contextSeenStartTlsEnabled || anyPropertyIsSet)) { Am Mi., 21. Okt. 2020 um 19:26 Uhr schrieb Bernd : > BTW: the security patch looks like "simple" is supposed to be rejected > when a principal is set, however this is not the case in my tests. Maybe > the method is not called correctly in this case? > > if ("simple".equalsIgnoreCase(authMechanism) && > !envprops.containsKey(SECURITY_PRINCIPAL)) { > > Gruss > Bernd > > Am Mi., 21. Okt. 2020 um 18:21 Uhr schrieb Bernd : > >> Hello, >> >> I am looking at 11.0.9 PSU (as of Zulu 11.43-sa) about the >> CVE-2020-14781 / JDK-8237990 fix and try to understand if my customers >> might be affected. >> >> jdk.jndi.ldap.mechsAllowedToSendCredentials >> >> It was not obvious to me, how the mechanism restriction works. >> >> According to Oracle and Redhat release notes it only looks at clear / >> non-TLS. >> >> - Can you confirm that SASL with wrapping is not considered as encrypted >> in this case? >> >> - Can you confirm it only applies to SASL based negotiation? (in my test >> SIMPLE with cleartext passwords works just fine) >> >> - Can you confirm it does not apply to "secure" mechanisms like >> DIGEST-MD5 or different methods like GSSAPI or SIMPLE? >> >> Gruss >> Bernd >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xuelei.fan at oracle.com Wed Oct 21 18:52:04 2020 From: xuelei.fan at oracle.com (Xuelei Fan) Date: Wed, 21 Oct 2020 11:52:04 -0700 Subject: RFR CSR: JDK-8254709 (Support for EdDSA signature scheme in JSSE) In-Reply-To: References: Message-ID: <32d6d895-5738-32c4-85d7-c89ff453f770@oracle.com> Hi Jamil, Sorry for delay. It took a few days before I was able to read the CSR. Just a few comments for your consideration. In the specification section, you mentioned how to disable the algorithms. It might not be necessary. It is just something we need to implement so that it does not break the mechanism. I'm not very sure of the order of EdDSA in the signature algorithms. EdDSA and EdDSA certificate are not popular yet. I'm fine with the order. Would you mind to share your consideration of the preference? The list of signature schemes is out of the quote box and hard to read. I may just list one scheme one line in one quote box, like: + ed25519 + ed448 ecdsa_secp256r1_sha256 I'm not very sure why EdDSA cannot apply to ServerKeyExchange and CertificateVerify in TLS 1.0 and 1.1. ServerKeyExchange and CertificateVerify is used to authenticate the server or the client's possession of the private key of the cert. So if EdDSA cannot be used for them, the EdDSA certificate should not be selected for TLS 1.0/1.1 as well. I did not read the RFC fully yet, it looks like that EdDSA can be used for TLS 1.0/1.1 ServerKeyExchange and CertificateVerify as well. I may miss something. Hope it helps. Xuelei On 10/13/2020 10:59 AM, Jamil Nimeh wrote: > Hi Folks, > > I just put out the draft CSR for the RFE that adds EdDSA support in > JSSE.? If anyone has some spare cycles to review this I'd appreciate it. > > https://bugs.openjdk.java.net/browse/JDK-8254709 > > Thanks, > > --Jamil > From psandoz at openjdk.java.net Wed Oct 21 19:08:25 2020 From: psandoz at openjdk.java.net (Paul Sandoz) Date: Wed, 21 Oct 2020 19:08:25 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v9] In-Reply-To: References: Message-ID: On Wed, 21 Oct 2020 11:33:27 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the first incubation round of the foreign linker access API incubation >> (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). >> >> The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. >> >> Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. >> >> A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). >> >> Thanks >> Maurizio >> >> Webrev: >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev >> >> Javadoc: >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html >> >> Specdiff (relative to [3]): >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254232 >> >> >> >> ### API Changes >> >> The API changes are actually rather slim: >> >> * `LibraryLookup` >> * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. >> * `FunctionDescriptor` >> * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. >> * `CLinker` >> * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. >> * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. >> * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. >> * `NativeScope` >> * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. >> * `MemorySegment` >> * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. >> >> ### Safety >> >> The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). >> >> ### Implementation changes >> >> The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). >> >> As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. >> >> Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. >> >> The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. >> >> This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. >> >> For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. >> >> A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. >> >> At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: >> >> * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. >> >> * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). >> >> * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. >> >> For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). >> >> Again, for more readings on the internals of the foreign linker support, please refer to [5]. >> >> #### Test changes >> >> Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. >> >> Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. >> >> [1] - https://openjdk.java.net/jeps/389 >> [2] - https://openjdk.java.net/jeps/393 >> [3] - https://git.openjdk.java.net/jdk/pull/548 >> [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md >> [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html > > Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 27 commits: > > - Merge branch 'master' into 8254231_linker > - Don't use JNI when generating native wrappers > > - Merge branch 'master' into 8254231_linker > - Fix incorrect capitalization in one copyright header > - Update copyright years, and add classpath exception to files that were missing it > - Use separate constants for native invoker code size > - Re-add file erroneously deleted (detected as rename) > - Re-add erroneously removed files > - Merge branch 'master' into 8254231_linker > > - Fix tests > - Fix more whitespaces > - ... and 17 more: https://git.openjdk.java.net/jdk/compare/da97ab5c...8c7b75da src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/AbstractNativeScope.java line 41: > 39: private static final int SCOPE_MASK = MemorySegment.READ | MemorySegment.WRITE; // no terminal operations allowed > 40: > 41: AbstractNativeScope(Thread ownerThread) { Since all concrete classes pass in `Thread.currentThread()` we can make this a no-arg constructor. The concrete classes can be marked as final? src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/AbstractNativeScope.java line 120: > 118: } > 119: } > 120: throw new AssertionError("Cannot get here!"); This code is a little confusing, effectively using an exception for control flow, within a retry loop. I recommend performing an explicit bounds check to determine if a new segment of `BLOCK_SIZE` is required from which to slice into. It will also be faster than the exceptional case. src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/CABI.java line 33: > 31: AArch64; > 32: > 33: public static CABI current() { I suspect this might be called often, create a private static final constant? src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/abi/Binding.java line 203: > 201: * -------------------- > 202: */ > 203: public abstract class Binding { Some design considerations, to consider later maybe. The IR representation could be simplified to use record classes (which should be exiting preview in 16), implementing a Binding interface. The interpreter and specializer (compiler) could be separate if need be, operating on a sequence of instructions that just hold the data. Pattern matching could be used on the binding instances. It may be simpler and more efficient if the compiler generated explicit byte code rather than using MH combinators. src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/abi/BindingInterpreter.java line 50: > 48: } > 49: > 50: interface StoreFunc { Annotate with `@FunctionalInterface`? src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/abi/CallingSequence.java line 51: > 49: } > 50: > 51: public Stream argBindings() { Duplicate methods `argBindings` and `argumentBindings`? src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/abi/CallingSequenceBuilder.java line 123: > 121: } > 122: > 123: private static final Set boxTags = EnumSet.of( s/boxTags/BOX_TAGS ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From psandoz at openjdk.java.net Wed Oct 21 19:08:22 2020 From: psandoz at openjdk.java.net (Paul Sandoz) Date: Wed, 21 Oct 2020 19:08:22 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: On Tue, 20 Oct 2020 17:23:26 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the first incubation round of the foreign linker access API incubation >> (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). >> >> The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. >> >> Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. >> >> A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). >> >> Thanks >> Maurizio >> >> Webrev: >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev >> >> Javadoc: >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html >> >> Specdiff (relative to [3]): >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254232 >> >> >> >> ### API Changes >> >> The API changes are actually rather slim: >> >> * `LibraryLookup` >> * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. >> * `FunctionDescriptor` >> * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. >> * `CLinker` >> * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. >> * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. >> * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. >> * `NativeScope` >> * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. >> * `MemorySegment` >> * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. >> >> ### Safety >> >> The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). >> >> ### Implementation changes >> >> The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). >> >> As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. >> >> Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. >> >> The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. >> >> This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. >> >> For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. >> >> A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. >> >> At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: >> >> * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. >> >> * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). >> >> * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. >> >> For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). >> >> Again, for more readings on the internals of the foreign linker support, please refer to [5]. >> >> #### Test changes >> >> Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. >> >> Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. >> >> [1] - https://openjdk.java.net/jeps/389 >> [2] - https://openjdk.java.net/jeps/393 >> [3] - https://git.openjdk.java.net/jdk/pull/548 >> [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md >> [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html > > Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 25 commits: > > - Merge branch 'master' into 8254231_linker > - Fix incorrect capitalization in one copyright header > - Update copyright years, and add classpath exception to files that were missing it > - Use separate constants for native invoker code size > - Re-add file erroneously deleted (detected as rename) > - Re-add erroneously removed files > - Merge branch 'master' into 8254231_linker > > - Fix tests > - Fix more whitespaces > - Fix whitespaces > - Remove rejected file > - ... and 15 more: https://git.openjdk.java.net/jdk/compare/cb6167b2...502bd980 Some of this is familiar to me from reviews in the `panama-foreign` repository, but less so than the memory API. I focused on the Java code, and ignored changes that are common with the memory API PR. If it helps in can provide a PR in the `panama-foreign` repository addressing editorial comments to the linker API. src/java.base/share/classes/java/lang/invoke/NativeMethodHandle.java line 36: > 34: import static java.lang.invoke.MethodHandleStatics.newInternalError; > 35: > 36: /** TODO */ Is the TODO to make this class public later and adjust the return type of `downcallHandle`? src/java.base/share/classes/java/lang/invoke/NativeMethodHandle.java line 145: > 143: */ > 144: private static class Lazy { > 145: static Class THIS_CLASS = NativeMethodHandle.class; final field? Is this field needed, as `NativeMethodHandle.class` could be used directly, or use a local variable instead in the static code block. src/java.base/share/classes/jdk/internal/access/JavaLangAccess.java line 46: > 44: import java.util.stream.Stream; > 45: > 46: import jdk.internal.loader.NativeLibrary; Unused import? src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 130: > 128: * @return the downcall method handle. > 129: * @throws IllegalArgumentException in the case of a carrier type and memory layout mismatch. > 130: */ Add `@see LibraryLookup#lookup` src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 126: > 124: * > 125: * @param symbol downcall symbol. > 126: * @param type the method type. s/method type/carrier type ? src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 129: > 127: * @param function the function descriptor. > 128: * @return the downcall method handle. > 129: * @throws IllegalArgumentException in the case of a carrier type and memory layout mismatch. carrier type and function descriptor mismatch? src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 139: > 137: * > 138: *

The returned segment is not thread-confined, and it only features > 139: * the {@link MemorySegment#CLOSE} access mode. When the returned segment is closed, Implying that it is shared? If so might be better to state that directly (with a link), and can be closed explicitly or left until can be collected by the GC? src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 145: > 143: * @param function the function descriptor. > 144: * @return the native stub segment. > 145: * @throws IllegalArgumentException in the case of a carrier type and memory layout mismatch. What's carrier type here? `target.type()`? "IllegalArgumentException if the target's method type the function descriptor mismatch" ? src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 201: > 199: } > 200: > 201: /** Extra spaces src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 212: > 210: * @param str the Java string to be converted into a C string. > 211: * @return a new native memory segment containing the converted C string. > 212: * @throws NullPointerException if either {@code str == null}. if {@code str == null}. src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 202: > 200: > 201: /** > 202: * Convert a Java string into a null-terminated C string, using the Converts (and in other places and for other methods) src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 314: > 312: * restricted methods, and use safe and supported functionalities, where possible. > 313: * @param addr the address at which the string is stored. > 314: * @param charset The {@linkplain java.nio.charset.Charset} to be used to compute the contents of the Java string. s/linkplain/link (and in other places) src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 352: > 350: * @param charset The {@linkplain java.nio.charset.Charset} to be used to compute the contents of the Java string. > 351: * @return a Java string with the contents of the null-terminated C string at given address. > 352: * @throws NullPointerException if {@code addr == null} or charset == null (and in other places) src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 380: > 378: > 379: /** > 380: * Allocate memory of given size using malloc. What if allocation failed? `OutOfMemoryError`? src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/FunctionDescriptor.java line 60: > 58: private FunctionDescriptor(MemoryLayout resLayout, Map attributes, MemoryLayout... argLayouts) { > 59: this.resLayout = resLayout; > 60: this.attributes = Collections.unmodifiableMap(attributes); Since `attributes` is never exposed directly or indirectly via a set of keys/values/entries there is no need to wrap it. src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/FunctionDescriptor.java line 100: > 98: /** > 99: * Returns the return layout associated with this function. > 100: * @return the return s/the return/the return layout (and other places) src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/FunctionDescriptor.java line 145: > 143: * @throws NullPointerException if any of the new argument layouts is null. > 144: */ > 145: public FunctionDescriptor appendArgumentLayouts(MemoryLayout... addedLayouts) { Might consider using "with" as in "withAppendedArgumentLayouts", "withReturnLayout", "withVoidReturnLayout" src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/LibraryLookup.java line 37: > 35: > 36: /** > 37: * A native library lookup. Exposes lookup operation for searching symbols, see {@link LibraryLookup#lookup(String)}. s/Exposes lookup/Exposes a lookup src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/LibraryLookup.java line 91: > 89: > 90: /** > 91: * Lookups a symbol with given name in this library. The returned symbol maintains a strong reference to this lookup object. s/Lookups/Looks up src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/LibraryLookup.java line 130: > 128: > 129: /** > 130: * Obtain a library lookup object corresponding to a library identified by given library name. Mention the context in which the library is found i.e. what ever the equivalent of LD_LIBRARY_PATH is in Java (the system property name escapes me at this moment). src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/NativeScope.java line 44: > 42: * by off-heap memory. Native scopes can be either bounded or unbounded, depending on whether the size > 43: * of the native scope is known statically. If an application knows before-hand how much memory it needs to allocate, > 44: * then using a bounded native scope will typically provide better performances than independently allocating the memory s/performances/performance src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/NativeScope.java line 54: > 52: * To allow for more usability, it is possible for a native scope to reclaim ownership of an existing memory segment > 53: * (see {@link MemorySegment#handoff(NativeScope)}). This might be useful to allow one or more segments which were independently > 54: * created to share the same life-cycle as a given native scope - which in turns enables client to group all memory s/enables client/enables a client src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/NativeScope.java line 85: > 83: * @return a segment for the newly allocated memory block. > 84: * @throws OutOfMemoryError if there is not enough space left in this native scope, that is, if > 85: * {@code limit() - size() < layout.byteSize()}. Where do the `limit` and `size` methods come from? src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/NativeScope.java line 374: > 372: * } > 373: * @param elementLayout the array element layout. > 374: * @param size the array element count. s/size/length or count? ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From kvn at openjdk.java.net Wed Oct 21 19:24:13 2020 From: kvn at openjdk.java.net (Vladimir Kozlov) Date: Wed, 21 Oct 2020 19:24:13 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v10] In-Reply-To: References: Message-ID: <06kvIM_3abB-35pPdgFfbvwCND6oe9QCBqXBQ8iIrZ4=.64ae7da4-be02-46cc-afde-ffeb9ec9d703@github.com> On Wed, 21 Oct 2020 09:19:57 GMT, Fei Yang wrote: > > Someone in Oracle have to run tier1-tier3 testing with these changes to make sure nothing is broken. I don't want to repeat 8254790. > > That's appreciated. > On my side, I run tier1-tier3 both on aarch64 linux and x86_64 linux. > The test result on these two platforms looks good for the latest changes. I started testing of 09: version. >> src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java line 604: >> >>> 602: add(ignore, "sun/security/provider/SHA5." + shaCompressName + "([BI)V"); >>> 603: } >>> 604: add(toBeInvestigated, "sun/security/provider/SHA3." + shaCompressName + "([BI)V"); >> >> This should be under `if (isJDK16OrHigher())` check. Something like this: >> https://github.com/openjdk/jdk/pull/650/files#diff-d1f378fc1b7fe041309e854d40b3a95a91e63fdecf0ecd9826b7c95eaeba314eR527 >> You can wait when Aleksey push it and update your changes > > OK. Will update with the following change after Aleksey's PR is integrated: > > --- a/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java > +++ b/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java > @@ -608,6 +608,10 @@ public class CheckGraalIntrinsics extends GraalTest { > if (!config.useSHA512Intrinsics()) { > add(ignore, "sun/security/provider/SHA5." + shaCompressName + "([BI)V"); > } > + > + if (isJDK16OrHigher()) { > + add(toBeInvestigated, "sun/security/provider/SHA3." + shaCompressName + "([BI)V"); > + } > } Yes, please, do that. ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From jamil.j.nimeh at oracle.com Wed Oct 21 20:01:44 2020 From: jamil.j.nimeh at oracle.com (Jamil Nimeh) Date: Wed, 21 Oct 2020 13:01:44 -0700 Subject: RFR CSR: JDK-8254709 (Support for EdDSA signature scheme in JSSE) In-Reply-To: <32d6d895-5738-32c4-85d7-c89ff453f770@oracle.com> References: <32d6d895-5738-32c4-85d7-c89ff453f770@oracle.com> Message-ID: <80a7419e-5df8-0619-97b8-ca296d9a355b@oracle.com> Hi Xuelei, thanks for the comments.? I'll respond in-line: On 10/21/2020 11:52 AM, Xuelei Fan wrote: > Hi Jamil, > > Sorry for delay.? It took a few days before I was able to read the > CSR. Just a few comments for your consideration. > > In the specification section, you mentioned how to disable the > algorithms. It might not be necessary.? It is just something we need > to implement so that it does not break the mechanism. JN: I was under the impression that where System/Security properties have an impact on the feature that we typically make note of it. > > I'm not very sure of the order of EdDSA in the signature algorithms. > EdDSA and EdDSA certificate are not popular yet.? I'm fine with the > order. Would you mind to share your consideration of the preference? No real consideration.? All I did was remove the explicit disable code in SignatureScheme.java:278 - 286.? The ordering in the enumeration then shows up in the signature_algorithms[_cert] extensions. Ed* schemes could be moved down a bit in the enumeration if we think that will make a difference.? I think it would only matter if a server or client had both ECC and EdDSA keyed certs that would be up for selection.? There are other implementations that do have Ed* signature schemes a little further down in their lists. > > The list of signature schemes is out of the quote box and hard to > read. I may just list one scheme one line in one quote box, like: > > ??? + ed25519 > ??? + ed448 > ????? ecdsa_secp256r1_sha256 JN: I could do that.? I was trying to avoid having it be a mile long, but if it's not rendering horizontally then I'll change it.? I suppose it doesn't have to be the list of all sig schemes.? Just a few lines of context on either side of the new signature schemes should drive the point home and keep it succinct. > > I'm not very sure why EdDSA cannot apply to ServerKeyExchange and > CertificateVerify in TLS 1.0 and 1.1. ServerKeyExchange and > CertificateVerify is used to authenticate the server or the client's > possession of the private key of the cert.? So if EdDSA cannot be used > for them, the EdDSA certificate should not be selected for TLS 1.0/1.1 > as well.? I did not read the RFC fully yet, it looks like that EdDSA > can be used for TLS 1.0/1.1 ServerKeyExchange and CertificateVerify as > well. ?I may miss something. JN: So far I have yet to find a server implementation that will accept a 1.0/1.1 client hello with no signature_algorithms extension and not barf.? This of course assumes that the server only has EdDSA keyed TLS certificates.? As I mentioned to you on the side, if it's RSA/DSA/ECC and has an EdDSA signature, then things work great.? To be fair, I haven't experimented with anything other than my modified JSSE and OpenSSL so far. > > Hope it helps. > > Xuelei > > On 10/13/2020 10:59 AM, Jamil Nimeh wrote: >> Hi Folks, >> >> I just put out the draft CSR for the RFE that adds EdDSA support in >> JSSE.? If anyone has some spare cycles to review this I'd appreciate it. >> >> https://bugs.openjdk.java.net/browse/JDK-8254709 >> >> Thanks, >> >> --Jamil >> From xuelei.fan at oracle.com Wed Oct 21 22:13:56 2020 From: xuelei.fan at oracle.com (Xuelei Fan) Date: Wed, 21 Oct 2020 15:13:56 -0700 Subject: RFR CSR: JDK-8254709 (Support for EdDSA signature scheme in JSSE) In-Reply-To: <80a7419e-5df8-0619-97b8-ca296d9a355b@oracle.com> References: <32d6d895-5738-32c4-85d7-c89ff453f770@oracle.com> <80a7419e-5df8-0619-97b8-ca296d9a355b@oracle.com> Message-ID: <9f6219ba-3b90-38f9-6956-ddaab8f89ece@oracle.com> On 10/21/2020 1:01 PM, Jamil Nimeh wrote: >> I'm not very sure why EdDSA cannot apply to ServerKeyExchange and >> CertificateVerify in TLS 1.0 and 1.1. ServerKeyExchange and >> CertificateVerify is used to authenticate the server or the client's >> possession of the private key of the cert.? So if EdDSA cannot be used >> for them, the EdDSA certificate should not be selected for TLS 1.0/1.1 >> as well.? I did not read the RFC fully yet, it looks like that EdDSA >> can be used for TLS 1.0/1.1 ServerKeyExchange and CertificateVerify as >> well. ?I may miss something. > JN: So far I have yet to find a server implementation that will accept a > 1.0/1.1 client hello with no signature_algorithms extension and not > barf. It's OK if we don't want to support EdDSA for TLS 1.0/1.1 for some reason. Although I would prefer to support for better interoperability. I did not get the idea of the CSR. It may be nice to have a explicit statement that we don't support certificates of EdDSA-capable public key for TLS 1.0 and 1.1. Xuelei From valeriep at openjdk.java.net Wed Oct 21 22:48:21 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 21 Oct 2020 22:48:21 GMT Subject: Integrated: 8199697: FIPS 186-4 RSA Key Generation In-Reply-To: References: Message-ID: <_jQlkvg1Mpf28GBz1ZRE7yUsaNFlEVMck_DG4EtpJ9Q=.b97a3a14-f113-4685-b45d-5b6d13fa9471@github.com> On Wed, 30 Sep 2020 03:25:06 GMT, Valerie Peng wrote: > Could someone please help review this RFE? Update existing RSA key pair generation code following the guidelines from FIPS 186-4 and FIPS 186-5 (draft). Current proposed changes updates the prime generation code (for P, Q) based on FIPS 186-4 B.3.3 when keysize and public exponent met the requirements set in FIPS 186-4/5. > > Thanks, > Valerie This pull request has now been integrated. Changeset: 1191a633 Author: Valerie Peng URL: https://git.openjdk.java.net/jdk/commit/1191a633 Stats: 176 lines in 2 files changed: 108 ins; 22 del; 46 mod 8199697: FIPS 186-4 RSA Key Generation Reviewed-by: xuelei ------------- PR: https://git.openjdk.java.net/jdk/pull/420 From fyang at openjdk.java.net Wed Oct 21 23:42:33 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Wed, 21 Oct 2020 23:42:33 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v11] In-Reply-To: References: Message-ID: > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. > Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 > > Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. > For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. > "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. > > We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. > Patch passed jtreg tier1-3 tests with QEMU system emulator. > Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make sure that there's no regression. > > We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java > We measured the performance benefit with an aarch64 cycle-accurate simulator. > Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. > > For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is detected. Fei Yang has updated the pull request incrementally with one additional commit since the last revision: Add if (isJDK16OrHigher()) check for SHA3 in CheckGraalIntrinsics.java ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/207/files - new: https://git.openjdk.java.net/jdk/pull/207/files/d32c8ad7..b43f9197 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=207&range=10 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=207&range=09-10 Stats: 4 lines in 1 file changed: 3 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/207.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/207/head:pull/207 PR: https://git.openjdk.java.net/jdk/pull/207 From fyang at openjdk.java.net Thu Oct 22 00:49:12 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Thu, 22 Oct 2020 00:49:12 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v10] In-Reply-To: <06kvIM_3abB-35pPdgFfbvwCND6oe9QCBqXBQ8iIrZ4=.64ae7da4-be02-46cc-afde-ffeb9ec9d703@github.com> References: <06kvIM_3abB-35pPdgFfbvwCND6oe9QCBqXBQ8iIrZ4=.64ae7da4-be02-46cc-afde-ffeb9ec9d703@github.com> Message-ID: On Wed, 21 Oct 2020 19:20:28 GMT, Vladimir Kozlov wrote: >> OK. Will update with the following change after Aleksey's PR is integrated: >> >> --- a/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java >> +++ b/src/jdk.internal.vm.compiler/share/classes/org.graalvm.compiler.hotspot.test/src/org/graalvm/compiler/hotspot/test/CheckGraalIntrinsics.java >> @@ -608,6 +608,10 @@ public class CheckGraalIntrinsics extends GraalTest { >> if (!config.useSHA512Intrinsics()) { >> add(ignore, "sun/security/provider/SHA5." + shaCompressName + "([BI)V"); >> } >> + >> + if (isJDK16OrHigher()) { >> + add(toBeInvestigated, "sun/security/provider/SHA3." + shaCompressName + "([BI)V"); >> + } >> } > > Yes, please, do that. Done. Commit: https://github.com/openjdk/jdk/pull/207/commits/b43f91970d44e6e0c1b3b4ef452ec388ecbecb83 I think this will not conflict with Aleksey's PR as we modify in different places of CheckGraalIntrinsics.java ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From kvn at openjdk.java.net Thu Oct 22 04:02:15 2020 From: kvn at openjdk.java.net (Vladimir Kozlov) Date: Thu, 22 Oct 2020 04:02:15 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v11] In-Reply-To: References: Message-ID: On Wed, 21 Oct 2020 23:42:33 GMT, Fei Yang wrote: >> Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com >> >> This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. >> Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 >> >> Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. >> For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. >> "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. >> >> We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. >> Patch passed jtreg tier1-3 tests with QEMU system emulator. >> Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make sure that there's no regression. >> >> We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java >> We measured the performance benefit with an aarch64 cycle-accurate simulator. >> Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. >> >> For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is detected. > > Fei Yang has updated the pull request incrementally with one additional commit since the last revision: > > Add if (isJDK16OrHigher()) check for SHA3 in CheckGraalIntrinsics.java tier1,2,3 passed. I verified that new SHA3 tests were run and passed. But because SHA3 is not enabled for now (even on aarch64), it does not test asm code. At least testing verified that changes in shared code does not cause any issues. ------------- Marked as reviewed by kvn (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/207 From fyang at openjdk.java.net Thu Oct 22 04:23:11 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Thu, 22 Oct 2020 04:23:11 GMT Subject: RFR: 8252204: AArch64: Implement SHA3 accelerator/intrinsic [v11] In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 03:59:45 GMT, Vladimir Kozlov wrote: > tier1,2,3 passed. I verified that new SHA3 tests were run and passed. > But because SHA3 is not enabled for now (even on aarch64), it does not test asm code. > At least testing verified that changes in shared code does not cause any issues. Great to hear that :-) Thanks for the effect. With that testing result and reviewing from three reviewers, I think it's safe to integrate. ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From fyang at openjdk.java.net Thu Oct 22 04:44:21 2020 From: fyang at openjdk.java.net (Fei Yang) Date: Thu, 22 Oct 2020 04:44:21 GMT Subject: Integrated: 8252204: AArch64: Implement SHA3 accelerator/intrinsic In-Reply-To: References: Message-ID: On Wed, 16 Sep 2020 16:36:54 GMT, Fei Yang wrote: > Contributed-by: ard.biesheuvel at linaro.org, dongbo4 at huawei.com > > This added an intrinsic for SHA3 using aarch64 v8.2 SHA3 Crypto Extensions. > Reference implementation for core SHA-3 transform using ARMv8.2 Crypto Extensions: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/crypto/sha3-ce-core.S?h=v5.4.52 > > Trivial adaptation in SHA3. implCompress is needed for the purpose of adding the intrinsic. > For SHA3, we need to pass one extra parameter "digestLength" to the stub for the calculation of block size. > "digestLength" is also used in for the EOR loop before keccak to differentiate different SHA3 variants. > > We added jtreg tests for SHA3 and used QEMU system emulator which supports SHA3 instructions to test the functionality. > Patch passed jtreg tier1-3 tests with QEMU system emulator. > Also verified with jtreg tier1-3 tests without SHA3 instructions on aarch64-linux-gnu and x86_64-linux-gnu, to make sure that there's no regression. > > We used one existing JMH test for performance test: test/micro/org/openjdk/bench/java/security/MessageDigests.java > We measured the performance benefit with an aarch64 cycle-accurate simulator. > Patch delivers 20% - 40% performance improvement depending on specific SHA3 digest length and size of the message. > > For now, this feature will not be enabled automatically for aarch64. We can auto-enable this when it is fully tested on real hardware. But for the above testing purposes, this is auto-enabled when the corresponding hardware feature is detected. This pull request has now been integrated. Changeset: b25d8940 Author: Fei Yang URL: https://git.openjdk.java.net/jdk/commit/b25d8940 Stats: 1265 lines in 36 files changed: 1010 ins; 22 del; 233 mod 8252204: AArch64: Implement SHA3 accelerator/intrinsic Co-authored-by: Ard Biesheuvel Co-authored-by: Dong Bo Reviewed-by: aph, kvn ------------- PR: https://git.openjdk.java.net/jdk/pull/207 From mcimadamore at openjdk.java.net Thu Oct 22 13:37:17 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 22 Oct 2020 13:37:17 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: On Tue, 20 Oct 2020 21:31:17 GMT, Paul Sandoz wrote: >> Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 25 commits: >> >> - Merge branch 'master' into 8254231_linker >> - Fix incorrect capitalization in one copyright header >> - Update copyright years, and add classpath exception to files that were missing it >> - Use separate constants for native invoker code size >> - Re-add file erroneously deleted (detected as rename) >> - Re-add erroneously removed files >> - Merge branch 'master' into 8254231_linker >> >> - Fix tests >> - Fix more whitespaces >> - Fix whitespaces >> - Remove rejected file >> - ... and 15 more: https://git.openjdk.java.net/jdk/compare/cb6167b2...502bd980 > > src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 126: > >> 124: * >> 125: * @param symbol downcall symbol. >> 126: * @param type the method type. > > s/method type/carrier type ? Not sure about this one? E.g. in my mental model, I often have seen "carrier type" associated with j.l.Class ? > src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 139: > >> 137: * >> 138: *

The returned segment is not thread-confined, and it only features >> 139: * the {@link MemorySegment#CLOSE} access mode. When the returned segment is closed, > > Implying that it is shared? If so might be better to state that directly (with a link), and can be closed explicitly or left until can be collected by the GC? `The returned segment is not thread-confined` ? Since it features CLOSE, it can be closed explicitly - I'm not sure 100% of what additional clarification is required - but I'm happy to make this clearer (I need more info). ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 22 14:29:14 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 22 Oct 2020 14:29:14 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v9] In-Reply-To: References: Message-ID: <1p5weKaRsQH1q9GBnZFqKPupXb-5hYfeMm-NxaPiPUM=.db7147ad-67e0-4738-9fa7-d1afdabe3705@github.com> On Wed, 21 Oct 2020 16:23:16 GMT, Paul Sandoz wrote: >> Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 27 commits: >> >> - Merge branch 'master' into 8254231_linker >> - Don't use JNI when generating native wrappers >> >> - Merge branch 'master' into 8254231_linker >> - Fix incorrect capitalization in one copyright header >> - Update copyright years, and add classpath exception to files that were missing it >> - Use separate constants for native invoker code size >> - Re-add file erroneously deleted (detected as rename) >> - Re-add erroneously removed files >> - Merge branch 'master' into 8254231_linker >> >> - Fix tests >> - Fix more whitespaces >> - ... and 17 more: https://git.openjdk.java.net/jdk/compare/da97ab5c...8c7b75da > > src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/AbstractNativeScope.java line 120: > >> 118: } >> 119: } >> 120: throw new AssertionError("Cannot get here!"); > > This code is a little confusing, effectively using an exception for control flow, within a retry loop. I recommend performing an explicit bounds check to determine if a new segment of `BLOCK_SIZE` is required from which to slice into. It will also be faster than the exceptional case. I hear you - that said, note that doing the bound check is not as trivial as it seems; you have to take into account the value of `sp` and add required alignment padding, and then perform a bound check against that - all logic which would need to be duplicated across the normal and the exceptional cases. Which is why the code settled the way it is. I'll see what I can do. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From jvernee at openjdk.java.net Thu Oct 22 14:34:19 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Thu, 22 Oct 2020 14:34:19 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: On Tue, 20 Oct 2020 21:08:26 GMT, Paul Sandoz wrote: >> Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 25 commits: >> >> - Merge branch 'master' into 8254231_linker >> - Fix incorrect capitalization in one copyright header >> - Update copyright years, and add classpath exception to files that were missing it >> - Use separate constants for native invoker code size >> - Re-add file erroneously deleted (detected as rename) >> - Re-add erroneously removed files >> - Merge branch 'master' into 8254231_linker >> >> - Fix tests >> - Fix more whitespaces >> - Fix whitespaces >> - Remove rejected file >> - ... and 15 more: https://git.openjdk.java.net/jdk/compare/cb6167b2...502bd980 > > src/java.base/share/classes/java/lang/invoke/NativeMethodHandle.java line 36: > >> 34: import static java.lang.invoke.MethodHandleStatics.newInternalError; >> 35: >> 36: /** TODO */ > > Is the TODO to make this class public later and adjust the return type of `downcallHandle`? IIRC this was added to silence a javac linter warning. Something should be added here. There is/was no plan to make this class public though. > src/java.base/share/classes/java/lang/invoke/NativeMethodHandle.java line 145: > >> 143: */ >> 144: private static class Lazy { >> 145: static Class THIS_CLASS = NativeMethodHandle.class; > > final field? Is this field needed, as `NativeMethodHandle.class` could be used directly, or use a local variable instead in the static code block. Yes, this was a leftover from old code. Can be a local var now, or remove altogether and replaced with `NativeMethodHandle.class` ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From jvernee at openjdk.java.net Thu Oct 22 14:39:18 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Thu, 22 Oct 2020 14:39:18 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: On Tue, 20 Oct 2020 21:53:55 GMT, Paul Sandoz wrote: >> Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 25 commits: >> >> - Merge branch 'master' into 8254231_linker >> - Fix incorrect capitalization in one copyright header >> - Update copyright years, and add classpath exception to files that were missing it >> - Use separate constants for native invoker code size >> - Re-add file erroneously deleted (detected as rename) >> - Re-add erroneously removed files >> - Merge branch 'master' into 8254231_linker >> >> - Fix tests >> - Fix more whitespaces >> - Fix whitespaces >> - Remove rejected file >> - ... and 15 more: https://git.openjdk.java.net/jdk/compare/cb6167b2...502bd980 > > src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/FunctionDescriptor.java line 60: > >> 58: private FunctionDescriptor(MemoryLayout resLayout, Map attributes, MemoryLayout... argLayouts) { >> 59: this.resLayout = resLayout; >> 60: this.attributes = Collections.unmodifiableMap(attributes); > > Since `attributes` is never exposed directly or indirectly via a set of keys/values/entries there is no need to wrap it. True. Though, it might be nice to keep like this as a bit of sanity checking. The map _should not_ be modified after construction. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From Justin.Cranford at entrust.com Thu Oct 22 03:44:13 2020 From: Justin.Cranford at entrust.com (Justin Cranford) Date: Thu, 22 Oct 2020 03:44:13 +0000 Subject: Please add HMAC keygen to SunPKCS11 Message-ID: Compare SunPKCS11 support for AES vs HMAC * AES => keygen is supported, and AES key can be used for encrypt and decrypt. * HMAC => keygen is not supported, but HMAC key can be used for MAC. This does not make sense. A third-party utility is required for HMAC keygen, but not for AES keygen. Use case: * PKCS#11 driver is v2.20. * This means AES-256-GCM is not available for confidentiality and integrity, because GCM supported was only added in PKCS#11 v2.40. * Fallback to AES-256-CBC and HmacSha256 is required for confidentiality and integrity, respectively. * Java can trigger AES keygen, but not HMAC keygen. A third-party utility is required to trigger HMAC keygen before running Java. Would it be possible to add the missing GENERIC-SECRET-KEY-GEN mechanism to SunPKCS11? Notice how that mechanism is missing from the documented SunPKCS11 algorithms and mechanisms. It is the same in Java 8 all the way up to 15. * https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html#ALG To reproduce and demonstrate the missing HMAC keygen issue, here is a small Java Maven project. * https://github.com/justincranford/pkcs11 The readme shows the commands to initialize the SoftHSM2 token, and use a third-party OpenSC utility to trigger HMAC keygen. It also shows how to set the required SoftHSM2 env variable and run the Maven build. The Maven build will execute the ITPkcs11.java integration test class. The tests demonstrate: * Successful SunPKCS11 login to SoftHSM2 and list any existing keys * Successful AES keygen, encrypt, decrypt * Successful HMAC mac * Failed HMAC keygen (because SunPKCS11 does not support GENERIC-SECRET-KEY-GEN mechanism yet) Thank you, Justin Cranford -------------- next part -------------- An HTML attachment was scrubbed... URL: From psandoz at openjdk.java.net Thu Oct 22 16:17:20 2020 From: psandoz at openjdk.java.net (Paul Sandoz) Date: Thu, 22 Oct 2020 16:17:20 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 13:30:13 GMT, Maurizio Cimadamore wrote: >> src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 126: >> >>> 124: * >>> 125: * @param symbol downcall symbol. >>> 126: * @param type the method type. >> >> s/method type/carrier type ? > > Not sure about this one? E.g. in my mental model, I often have seen "carrier type" associated with j.l.Class ? Ah, i see, i find it confusing that "carrier type" is mentioned in the `@throws`, and was assuming it was an alias for method type, did you really mean method type? >> src/jdk.incubator.foreign/share/classes/jdk/incubator/foreign/CLinker.java line 139: >> >>> 137: * >>> 138: *

The returned segment is not thread-confined, and it only features >>> 139: * the {@link MemorySegment#CLOSE} access mode. When the returned segment is closed, >> >> Implying that it is shared? If so might be better to state that directly (with a link), and can be closed explicitly or left until can be collected by the GC? > > `The returned segment is not thread-confined` ? Since it features CLOSE, it can be closed explicitly - I'm not sure 100% of what additional clarification is required - but I'm happy to make this clearer (I need more info). Sometimes it's clearer to state the non-negative term i.e. _shared_ which is now something more explicit e.g. > The returned segment is _shared_ [add link?] (not thread-confined) That is really what i was trying to get at, rather than the CLOSE+GC aspects. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From psandoz at openjdk.java.net Thu Oct 22 16:17:19 2020 From: psandoz at openjdk.java.net (Paul Sandoz) Date: Thu, 22 Oct 2020 16:17:19 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 14:31:12 GMT, Jorn Vernee wrote: >> src/java.base/share/classes/java/lang/invoke/NativeMethodHandle.java line 36: >> >>> 34: import static java.lang.invoke.MethodHandleStatics.newInternalError; >>> 35: >>> 36: /** TODO */ >> >> Is the TODO to make this class public later and adjust the return type of `downcallHandle`? > > IIRC this was added to silence a javac linter warning. Something should be added here. There is/was no plan to make this class public though. It's odd the lint warning is triggering on a package private class and private methods. Separately, I recommend updating `make/CompileJavaModules.gmk` and adding `-Xdoclint:all/protected` for the module (i recently did this for the vector API see [here](https://github.com/openjdk/jdk/commit/000143504408ac7938e9f493c17c4dbb994045f9#diff-118e609b9974c0ce8af7950711461c7ab4620c9d4f4c99d231f598696f8e05d0) ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From psandoz at openjdk.java.net Thu Oct 22 16:17:21 2020 From: psandoz at openjdk.java.net (Paul Sandoz) Date: Thu, 22 Oct 2020 16:17:21 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v9] In-Reply-To: <1p5weKaRsQH1q9GBnZFqKPupXb-5hYfeMm-NxaPiPUM=.db7147ad-67e0-4738-9fa7-d1afdabe3705@github.com> References: <1p5weKaRsQH1q9GBnZFqKPupXb-5hYfeMm-NxaPiPUM=.db7147ad-67e0-4738-9fa7-d1afdabe3705@github.com> Message-ID: <4LF0LKSTD3ZkvV2kWKDqqeDAxh0mxMaB7eH2hakq7O4=.34d04357-f168-4ae2-b224-212122970e7d@github.com> On Thu, 22 Oct 2020 14:26:37 GMT, Maurizio Cimadamore wrote: >> src/jdk.incubator.foreign/share/classes/jdk/internal/foreign/AbstractNativeScope.java line 120: >> >>> 118: } >>> 119: } >>> 120: throw new AssertionError("Cannot get here!"); >> >> This code is a little confusing, effectively using an exception for control flow, within a retry loop. I recommend performing an explicit bounds check to determine if a new segment of `BLOCK_SIZE` is required from which to slice into. It will also be faster than the exceptional case. > > I hear you - that said, note that doing the bound check is not as trivial as it seems; you have to take into account the value of `sp` and add required alignment padding, and then perform a bound check against that - all logic which would need to be duplicated across the normal and the exceptional cases. Which is why the code settled the way it is. I'll see what I can do. Yeah, i would have probably done the same thing initially. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 22 16:34:28 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 22 Oct 2020 16:34:28 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: On Wed, 21 Oct 2020 19:05:42 GMT, Paul Sandoz wrote: >> Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 25 commits: >> >> - Merge branch 'master' into 8254231_linker >> - Fix incorrect capitalization in one copyright header >> - Update copyright years, and add classpath exception to files that were missing it >> - Use separate constants for native invoker code size >> - Re-add file erroneously deleted (detected as rename) >> - Re-add erroneously removed files >> - Merge branch 'master' into 8254231_linker >> >> - Fix tests >> - Fix more whitespaces >> - Fix whitespaces >> - Remove rejected file >> - ... and 15 more: https://git.openjdk.java.net/jdk/compare/cb6167b2...502bd980 > > Some of this is familiar to me from reviews in the `panama-foreign` repository, but less so than the memory API. I focused on the Java code, and ignored changes that are common with the memory API PR. > > If it helps in can provide a PR in the `panama-foreign` repository addressing editorial comments to the linker API. I've just uploaded a new iteration which addresses most of @PaulSandoz comments on the API/Java impl. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 22 16:34:27 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 22 Oct 2020 16:34:27 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v10] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). > > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. > > Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. > > A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). > > Thanks > Maurizio > > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). > > ### Implementation changes > > The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). > > As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. > > Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. > > The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. > > This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. > > For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. > > A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. > > At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: > > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). > > Again, for more readings on the internals of the foreign linker support, please refer to [5]. > > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. > > Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. > > [1] - https://openjdk.java.net/jeps/389 > [2] - https://openjdk.java.net/jeps/393 > [3] - https://git.openjdk.java.net/jdk/pull/548 > [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md > [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 39 commits: - Merge pull request #4 from JornVernee/Missing_ResourceMarks Add missing resource marks before parsing descriptors - Add missing resource marks before parsing descriptors - Fix CLinker javadoc - Simplify AbstractNativeScope::allocate - Fix most review comments Fix Graal intrinsics test failure - And more copyright fixes - Fix more copyright headers - Fix copyright of AbstractNativeScope - Fix aarch issues - Remove spurious include - ... and 29 more: https://git.openjdk.java.net/jdk/compare/cc50c8d4...21f50872 ------------- Changes: https://git.openjdk.java.net/jdk/pull/634/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=09 Stats: 75773 lines in 270 files changed: 72888 ins; 1608 del; 1277 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 22 16:34:28 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 22 Oct 2020 16:34:28 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: <903tobsjq6ao1evfv2CXzPE9jLfRWcVXrghBViPU4eo=.01d1f2d9-6c6e-456c-950b-632dd6619e1b@github.com> On Thu, 22 Oct 2020 15:46:33 GMT, Paul Sandoz wrote: >> IIRC this was added to silence a javac linter warning. Something should be added here. There is/was no plan to make this class public though. > > It's odd the lint warning is triggering on a package private class and private methods. Separately, I recommend updating `make/CompileJavaModules.gmk` and adding `-Xdoclint:all/protected` for the module (i recently did this for the vector API see [here](https://github.com/openjdk/jdk/commit/000143504408ac7938e9f493c17c4dbb994045f9#diff-118e609b9974c0ce8af7950711461c7ab4620c9d4f4c99d231f598696f8e05d0) There's no lint warning for private method, a fix is coming. As for makefile changes, I'd prefer to postpone after integration, or add them to the foreign memory access PR, since that's not specific to this PR. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 22 16:34:29 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 22 Oct 2020 16:34:29 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v9] In-Reply-To: References: Message-ID: <729-YxJZTBRSEYbheYgTYKRo2FpSlyaPCrCSfI0i67Y=.5445d130-fa23-46f3-b8ab-86ba8fafeaee@github.com> On Wed, 21 Oct 2020 17:42:53 GMT, Paul Sandoz wrote: > Some design considerations, to consider later maybe. > > The IR representation could be simplified to use record classes (which should be exiting preview in 16), implementing a Binding interface. The interpreter and specializer (compiler) could be separate if need be, operating on a sequence of instructions that just hold the data. Pattern matching could be used on the binding instances. It may be simpler and more efficient if the compiler generated explicit byte code rather than using MH combinators. Good suggestions - I think this is good to consider post-integration. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 22 16:57:30 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 22 Oct 2020 16:57:30 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v11] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). > > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. > > Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. > > A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). > > Thanks > Maurizio > > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). > > ### Implementation changes > > The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). > > As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. > > Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. > > The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. > > This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. > > For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. > > A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. > > At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: > > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). > > Again, for more readings on the internals of the foreign linker support, please refer to [5]. > > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. > > Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. > > [1] - https://openjdk.java.net/jeps/389 > [2] - https://openjdk.java.net/jeps/393 > [3] - https://git.openjdk.java.net/jdk/pull/548 > [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md > [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request incrementally with three additional commits since the last revision: - Merge pull request #2 from JornVernee/No_JNI_Comments_v2 Address Coleen's review comments on "Don't use JNI when generating native wrappers" - Merge branch '8254231_linker' into No_JNI_Comments_v2 - Review comments: - Use TempNewSymbol - JVM_* -> JNI_* - rename parseXX -> parse_x_x - remove ref to global function using '::' - make 2 functions static, instead of ForeignGlobals class members ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/634/files - new: https://git.openjdk.java.net/jdk/pull/634/files/21f50872..0c892293 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=10 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=09-10 Stats: 64 lines in 12 files changed: 6 ins; 20 del; 38 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From jvernee at openjdk.java.net Thu Oct 22 16:57:31 2020 From: jvernee at openjdk.java.net (Jorn Vernee) Date: Thu, 22 Oct 2020 16:57:31 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v7] In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 16:31:00 GMT, Maurizio Cimadamore wrote: >> Some of this is familiar to me from reviews in the `panama-foreign` repository, but less so than the memory API. I focused on the Java code, and ignored changes that are common with the memory API PR. >> >> If it helps in can provide a PR in the `panama-foreign` repository addressing editorial comments to the linker API. > > I've just uploaded a new iteration which addresses most of @PaulSandoz comments on the API/Java impl. The most recent changes address Coleen's review comments on: #1 The changes I've made are: - Use TempNewSymbol when creating a symbol in field_offset & find_InstanceKlass - change JVM_* macros -> JNI_* - rename parseABIDescritpor & parseBufferLayout -> parse_abi_descriptor & parse_buffer_layout - remove ref to global function using '::', by merging the 2 functions together - make 2 functions static, instead of ForeignGlobals class members ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Thu Oct 22 17:04:34 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 22 Oct 2020 17:04:34 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v12] In-Reply-To: References: Message-ID: <2IKx6cpc-IGP3jZtr0s2I14BWM6ptyFD26szPl3b1ng=.9d956b98-dfe6-4a45-a371-bf86923214fb@github.com> > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). > > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. > > Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. > > A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). > > Thanks > Maurizio > > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). > > ### Implementation changes > > The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). > > As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. > > Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. > > The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. > > This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. > > For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. > > A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. > > At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: > > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). > > Again, for more readings on the internals of the foreign linker support, please refer to [5]. > > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. > > Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. > > [1] - https://openjdk.java.net/jeps/389 > [2] - https://openjdk.java.net/jeps/393 > [3] - https://git.openjdk.java.net/jdk/pull/548 > [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md > [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Fix whitespaces ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/634/files - new: https://git.openjdk.java.net/jdk/pull/634/files/0c892293..2c2d2a70 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=11 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=10-11 Stats: 2 lines in 2 files changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From jjg at openjdk.java.net Thu Oct 22 17:29:21 2020 From: jjg at openjdk.java.net (Jonathan Gibbons) Date: Thu, 22 Oct 2020 17:29:21 GMT Subject: RFR: JDK-8255262: Remove use of legacy custom @spec tag Message-ID: The change is (just) to remove legacy usages of a JDK-private custom tag. ------------- Commit messages: - JDK-8255262: Remove use of legacy custom @spec tag Changes: https://git.openjdk.java.net/jdk/pull/814/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=814&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8255262 Stats: 209 lines in 69 files changed: 0 ins; 209 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/814.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/814/head:pull/814 PR: https://git.openjdk.java.net/jdk/pull/814 From lancea at openjdk.java.net Thu Oct 22 17:35:15 2020 From: lancea at openjdk.java.net (Lance Andersen) Date: Thu, 22 Oct 2020 17:35:15 GMT Subject: RFR: JDK-8255262: Remove use of legacy custom @spec tag In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 17:16:23 GMT, Jonathan Gibbons wrote: > The change is (just) to remove legacy usages of a JDK-private custom tag. looks fine ------------- Marked as reviewed by lancea (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/814 From iris at openjdk.java.net Thu Oct 22 17:46:16 2020 From: iris at openjdk.java.net (Iris Clark) Date: Thu, 22 Oct 2020 17:46:16 GMT Subject: RFR: JDK-8255262: Remove use of legacy custom @spec tag In-Reply-To: References: Message-ID: <-NCS5lF0mdeaDq1CyQYTrh0gGVozPafltjNYsEpO488=.d795b2ec-acc2-4c06-8434-757b2095e386@github.com> On Thu, 22 Oct 2020 17:16:23 GMT, Jonathan Gibbons wrote: > The change is (just) to remove legacy usages of a JDK-private custom tag. Nice clean-up. ------------- Marked as reviewed by iris (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/814 From mr at openjdk.java.net Thu Oct 22 17:46:15 2020 From: mr at openjdk.java.net (Mark Reinhold) Date: Thu, 22 Oct 2020 17:46:15 GMT Subject: RFR: JDK-8255262: Remove use of legacy custom @spec tag In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 17:16:23 GMT, Jonathan Gibbons wrote: > The change is (just) to remove legacy usages of a JDK-private custom tag. As the creator of these tags many moons ago, I approve this change. ------------- Marked as reviewed by mr (Lead). PR: https://git.openjdk.java.net/jdk/pull/814 From alanb at openjdk.java.net Thu Oct 22 17:46:17 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Thu, 22 Oct 2020 17:46:17 GMT Subject: RFR: JDK-8255262: Remove use of legacy custom @spec tag In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 17:16:23 GMT, Jonathan Gibbons wrote: > The change is (just) to remove legacy usages of a JDK-private custom tag. Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/814 From darcy at openjdk.java.net Thu Oct 22 17:46:18 2020 From: darcy at openjdk.java.net (Joe Darcy) Date: Thu, 22 Oct 2020 17:46:18 GMT Subject: RFR: JDK-8255262: Remove use of legacy custom @spec tag In-Reply-To: References: Message-ID: <5hlum1g6yK7pF3TkDMBvYSlK0Ca3bVh1VM3XJUVAvCk=.93e60ef2-400c-4775-a4c5-1f290e14ed57@github.com> On Thu, 22 Oct 2020 17:16:23 GMT, Jonathan Gibbons wrote: > The change is (just) to remove legacy usages of a JDK-private custom tag. Marked as reviewed by darcy (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/814 From mchung at openjdk.java.net Thu Oct 22 17:56:16 2020 From: mchung at openjdk.java.net (Mandy Chung) Date: Thu, 22 Oct 2020 17:56:16 GMT Subject: RFR: JDK-8255262: Remove use of legacy custom @spec tag In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 17:16:23 GMT, Jonathan Gibbons wrote: > The change is (just) to remove legacy usages of a JDK-private custom tag. Marked as reviewed by mchung (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/814 From ascarpino at openjdk.java.net Thu Oct 22 18:05:21 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 22 Oct 2020 18:05:21 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <2udH_wWPCcqka4WmVmcLlezitoprq0c_w9YHxAbmP1E=.975d839f-ce4f-4056-9487-a3233e758331@github.com> References: <2udH_wWPCcqka4WmVmcLlezitoprq0c_w9YHxAbmP1E=.975d839f-ce4f-4056-9487-a3233e758331@github.com> Message-ID: On Mon, 12 Oct 2020 18:45:28 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GCTR.java line 237: > >> 235: encrypt(in, offset, processed, out, 0); >> 236: dst.get(out, 0, processed); >> 237: return len; > > This block of code looks strange? > len = inLen % AES_BLOCK_SIZE => len must be between 0...AES_BLOCK_SIZE-1, I think you meant to use len = (inLen - inLen% AES_BLOCK_SIZE) > dst.get(...) should be dst.put(...) Ok. Not surprising no test failed given it's a rollover situation ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Thu Oct 22 18:08:19 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 22 Oct 2020 18:08:19 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <2udH_wWPCcqka4WmVmcLlezitoprq0c_w9YHxAbmP1E=.975d839f-ce4f-4056-9487-a3233e758331@github.com> References: <2udH_wWPCcqka4WmVmcLlezitoprq0c_w9YHxAbmP1E=.975d839f-ce4f-4056-9487-a3233e758331@github.com> Message-ID: <-g-1okNkxejVVG0mckPLUS90L5CgVvhTNCOQqWjeZpc=.adcded37-6ddc-4fd4-8374-62f2702ce790@github.com> On Mon, 12 Oct 2020 18:26:14 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GCTR.java line 165: > >> 163: } >> 164: >> 165: int update(ByteBuffer src, ByteBuffer dst) { > > Based on the impl in GaloisCounterMode, this is only called when src.remaining() >= 128. Perhaps documenting the conditions here as there are no checks on the src/dst sizes as in the byte[] case. Sounds good ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Thu Oct 22 18:15:16 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 22 Oct 2020 18:15:16 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Mon, 12 Oct 2020 17:35:48 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GCTR.java line 146: > >> 144: private static final int MAX_LEN = 1024; >> 145: >> 146: int doFinal(ByteBuffer src, ByteBuffer dst) throws IllegalBlockSizeException { > > Conventionally, we put doFinal() method after update() method. Perhaps can we reorder it in the same way? ok ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From jjg at openjdk.java.net Thu Oct 22 19:49:22 2020 From: jjg at openjdk.java.net (Jonathan Gibbons) Date: Thu, 22 Oct 2020 19:49:22 GMT Subject: Integrated: JDK-8255262: Remove use of legacy custom @spec tag In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 17:16:23 GMT, Jonathan Gibbons wrote: > The change is (just) to remove legacy usages of a JDK-private custom tag. This pull request has now been integrated. Changeset: 0aa3c925 Author: Jonathan Gibbons URL: https://git.openjdk.java.net/jdk/commit/0aa3c925 Stats: 209 lines in 69 files changed: 0 ins; 209 del; 0 mod 8255262: Remove use of legacy custom @spec tag Reviewed-by: lancea, mr, iris, alanb, darcy, mchung ------------- PR: https://git.openjdk.java.net/jdk/pull/814 From ascarpino at openjdk.java.net Thu Oct 22 20:11:16 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 22 Oct 2020 20:11:16 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> References: <96q6DcIMcJGYMYIgS5NYiyY3dgVEHK2Op9fATwnZ4Zs=.f92c0b55-dcf2-452a-9f8c-eba504e2ce11@github.com> Message-ID: On Fri, 9 Oct 2020 02:38:05 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> Xuelei comments > > src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 614: > >> 612: } >> 613: >> 614: checkDataLength(processed, len); > > It seems that both checks (line 605 and 614) can be combined into: > checkDataLength(processed, Math.addExact(len, tagLenBytes)); I changed both encryptFinal for byte[] and bytebuffer, as I got that check from copying the byte[] method. They do look like mostly redundant checks, and I moved them to the top. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Thu Oct 22 20:14:21 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Thu, 22 Oct 2020 20:14:21 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v2] In-Reply-To: References: Message-ID: On Thu, 8 Oct 2020 00:13:37 GMT, Anthony Scarpino wrote: >> src/java.base/share/classes/com/sun/crypto/provider/CipherCore.java line 939: >> >>> 937: // if the size of specified output buffer is less than >>> 938: // the length of the cipher text, then the current >>> 939: // content of cipher has to be preserved in order for >> >> This change is somewhat dangerous. When using the supplied output buffer directly, you may corrupt its content w/ padding bytes. This optimization should only be applied when no padding is involved. In addition, input and output can point to the same buffer with all sorts of index combinations, i.e. inOfs == outOfs, inOfs < outOfs, inOfs > outOfs. With "outWithPadding" approach, no need to check. However, for "internalOutput", data corruption may happen. This kind of problem can be very hard to diagnose. Have to be very very careful here as this code may impact all ciphers... > > - I do not understand where the corruption comes from. The user provides a buffer that output is suppose to be placed into, what could it be corrupting? The existing tests (SameBuffer, in particular) works fine with this and the ByteBuffer calls. I spent a lot of time trying to get those same buffer tests to pass. > - It was my expectation that checkOutputCapacity() is making sure all the inOfs ==<> outOfs are going to work. Does that not catch all cases? > - outWithPadding" is excessive because it doubles the allocation for every operation followed by a copy to the original buffer, even if the original buffer was adequate. I'm ok with doing the extra alloc & copy in certain situations, but not everytime. Can you be more specific what things may fail that we don't already check for in the regression tests? I wrote a whole new tests to exercise all the buffers with and without offsets. Hopefully that covers the concern. The test found 4 bugs.. ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From redestad at openjdk.java.net Fri Oct 23 08:03:51 2020 From: redestad at openjdk.java.net (Claes Redestad) Date: Fri, 23 Oct 2020 08:03:51 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 20:46:15 GMT, ?????? ??????? wrote: > As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: > @State(Scope.Thread) > @OutputTimeUnit(TimeUnit.NANOSECONDS) > @BenchmarkMode(value = Mode.AverageTime) > public class AtomicBenchmark { > @Benchmark > public Object defaultValue() { > return new AtomicInteger(); > } > @Benchmark > public Object explicitValue() { > return new AtomicInteger(0); > } > } > THis benchmark demonstrates that `explicitValue()` is much slower: > Benchmark Mode Cnt Score Error Units > AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op > AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op > So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. > > I've tested the changes locally, both tier1 and tier 2 are ok. > > Could one create an issue for tracking this? Filed [8255299](https://bugs.openjdk.java.net/browse/JDK-8255299) for this. Prefix the name of the PR with "8255299: " and it should pass checks. ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From github.com+10835776+stsypanov at openjdk.java.net Fri Oct 23 08:03:51 2020 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Fri, 23 Oct 2020 08:03:51 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects Message-ID: As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: @State(Scope.Thread) @OutputTimeUnit(TimeUnit.NANOSECONDS) @BenchmarkMode(value = Mode.AverageTime) public class AtomicBenchmark { @Benchmark public Object defaultValue() { return new AtomicInteger(); } @Benchmark public Object explicitValue() { return new AtomicInteger(0); } } THis benchmark demonstrates that `explicitValue()` is much slower: Benchmark Mode Cnt Score Error Units AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. I've tested the changes locally, both tier1 and tier 2 are ok. Could one create an issue for tracking this? ------------- Commit messages: - 8255299: Drop explicit zeroing at instantiation of Atomic* objects Changes: https://git.openjdk.java.net/jdk/pull/818/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=818&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8255299 Stats: 19 lines in 17 files changed: 0 ins; 3 del; 16 mod Patch: https://git.openjdk.java.net/jdk/pull/818.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/818/head:pull/818 PR: https://git.openjdk.java.net/jdk/pull/818 From redestad at openjdk.java.net Fri Oct 23 08:08:35 2020 From: redestad at openjdk.java.net (Claes Redestad) Date: Fri, 23 Oct 2020 08:08:35 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 20:46:15 GMT, ?????? ??????? wrote: > As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: > @State(Scope.Thread) > @OutputTimeUnit(TimeUnit.NANOSECONDS) > @BenchmarkMode(value = Mode.AverageTime) > public class AtomicBenchmark { > @Benchmark > public Object defaultValue() { > return new AtomicInteger(); > } > @Benchmark > public Object explicitValue() { > return new AtomicInteger(0); > } > } > THis benchmark demonstrates that `explicitValue()` is much slower: > Benchmark Mode Cnt Score Error Units > AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op > AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op > So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. > > I've tested the changes locally, both tier1 and tier 2 are ok. > > Could one create an issue for tracking this? Marked as reviewed by redestad (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From serb at openjdk.java.net Fri Oct 23 08:17:40 2020 From: serb at openjdk.java.net (Sergey Bylokhov) Date: Fri, 23 Oct 2020 08:17:40 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 20:46:15 GMT, ?????? ??????? wrote: > As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: > @State(Scope.Thread) > @OutputTimeUnit(TimeUnit.NANOSECONDS) > @BenchmarkMode(value = Mode.AverageTime) > public class AtomicBenchmark { > @Benchmark > public Object defaultValue() { > return new AtomicInteger(); > } > @Benchmark > public Object explicitValue() { > return new AtomicInteger(0); > } > } > THis benchmark demonstrates that `explicitValue()` is much slower: > Benchmark Mode Cnt Score Error Units > AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op > AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op > So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. > > I've tested the changes locally, both tier1 and tier 2 are ok. > > Could one create an issue for tracking this? The changes in src/java.desktop looks fine. ------------- Marked as reviewed by serb (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/818 From dfuchs at openjdk.java.net Fri Oct 23 09:14:43 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Fri, 23 Oct 2020 09:14:43 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 20:46:15 GMT, ?????? ??????? wrote: > As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: > @State(Scope.Thread) > @OutputTimeUnit(TimeUnit.NANOSECONDS) > @BenchmarkMode(value = Mode.AverageTime) > public class AtomicBenchmark { > @Benchmark > public Object defaultValue() { > return new AtomicInteger(); > } > @Benchmark > public Object explicitValue() { > return new AtomicInteger(0); > } > } > THis benchmark demonstrates that `explicitValue()` is much slower: > Benchmark Mode Cnt Score Error Units > AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op > AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op > So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. > > I've tested the changes locally, both tier1 and tier 2 are ok. > > Could one create an issue for tracking this? src/java.base/share/classes/sun/net/ResourceManager.java line 65: > 63: } catch (NumberFormatException e) {} > 64: maxSockets = defmax; > 65: numSockets = new AtomicInteger(); Changes in sun/net look good to me. ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From dfuchs at openjdk.java.net Fri Oct 23 09:17:36 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Fri, 23 Oct 2020 09:17:36 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Fri, 23 Oct 2020 08:15:00 GMT, Sergey Bylokhov wrote: >> As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: >> @State(Scope.Thread) >> @OutputTimeUnit(TimeUnit.NANOSECONDS) >> @BenchmarkMode(value = Mode.AverageTime) >> public class AtomicBenchmark { >> @Benchmark >> public Object defaultValue() { >> return new AtomicInteger(); >> } >> @Benchmark >> public Object explicitValue() { >> return new AtomicInteger(0); >> } >> } >> THis benchmark demonstrates that `explicitValue()` is much slower: >> Benchmark Mode Cnt Score Error Units >> AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op >> AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op >> So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. >> >> I've tested the changes locally, both tier1 and tier 2 are ok. >> >> Could one create an issue for tracking this? > > The changes in src/java.desktop looks fine. Changes to `java.logging` and `java.net.http` also look good to me. ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From github.com+10835776+stsypanov at openjdk.java.net Fri Oct 23 09:59:41 2020 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Fri, 23 Oct 2020 09:59:41 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Fri, 23 Oct 2020 09:12:15 GMT, Daniel Fuchs wrote: >> As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: >> @State(Scope.Thread) >> @OutputTimeUnit(TimeUnit.NANOSECONDS) >> @BenchmarkMode(value = Mode.AverageTime) >> public class AtomicBenchmark { >> @Benchmark >> public Object defaultValue() { >> return new AtomicInteger(); >> } >> @Benchmark >> public Object explicitValue() { >> return new AtomicInteger(0); >> } >> } >> THis benchmark demonstrates that `explicitValue()` is much slower: >> Benchmark Mode Cnt Score Error Units >> AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op >> AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op >> So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. >> >> I've tested the changes locally, both tier1 and tier 2 are ok. >> >> Could one create an issue for tracking this? > > src/java.base/share/classes/sun/net/ResourceManager.java line 65: > >> 63: } catch (NumberFormatException e) {} >> 64: maxSockets = defmax; >> 65: numSockets = new AtomicInteger(); > > Changes in sun/net look good to me. @dfuch Could you then sponsor this PR? ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From ihse at openjdk.java.net Fri Oct 23 11:04:41 2020 From: ihse at openjdk.java.net (Magnus Ihse Bursie) Date: Fri, 23 Oct 2020 11:04:41 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v12] In-Reply-To: <2IKx6cpc-IGP3jZtr0s2I14BWM6ptyFD26szPl3b1ng=.9d956b98-dfe6-4a45-a371-bf86923214fb@github.com> References: <2IKx6cpc-IGP3jZtr0s2I14BWM6ptyFD26szPl3b1ng=.9d956b98-dfe6-4a45-a371-bf86923214fb@github.com> Message-ID: On Thu, 22 Oct 2020 17:04:34 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the first incubation round of the foreign linker access API incubation >> (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). >> >> The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. >> >> Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. >> >> A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). >> >> Thanks >> Maurizio >> >> Webrev: >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev >> >> Javadoc: >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html >> >> Specdiff (relative to [3]): >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254232 >> >> >> >> ### API Changes >> >> The API changes are actually rather slim: >> >> * `LibraryLookup` >> * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. >> * `FunctionDescriptor` >> * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. >> * `CLinker` >> * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. >> * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. >> * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. >> * `NativeScope` >> * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. >> * `MemorySegment` >> * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. >> >> ### Safety >> >> The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). >> >> ### Implementation changes >> >> The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). >> >> As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. >> >> Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. >> >> The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. >> >> This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. >> >> For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. >> >> A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. >> >> At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: >> >> * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. >> >> * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). >> >> * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. >> >> For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). >> >> Again, for more readings on the internals of the foreign linker support, please refer to [5]. >> >> #### Test changes >> >> Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. >> >> Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. >> >> [1] - https://openjdk.java.net/jeps/389 >> [2] - https://openjdk.java.net/jeps/393 >> [3] - https://git.openjdk.java.net/jdk/pull/548 >> [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md >> [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Fix whitespaces Changes requested by ihse (Reviewer). make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 148: > 146: > 147: $(DEST): $(BUILD_TOOLS_JDK) $(SCOPED_MEMORY_ACCESS_TEMPLATE) $(SCOPED_MEMORY_ACCESS_BIN_TEMPLATE) > 148: $(MKDIR) -p $(SCOPED_MEMORY_ACCESS_GENSRC_DIR) Please use `$(call MakeDir, $(SCOPED_MEMORY_ACCESS_GENSRC_DIR))` instead. make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 34: > 32: SCOPED_MEMORY_ACCESS_TEMPLATE := $(SCOPED_MEMORY_ACCESS_SRC_DIR)/X-ScopedMemoryAccess.java.template > 33: SCOPED_MEMORY_ACCESS_BIN_TEMPLATE := $(SCOPED_MEMORY_ACCESS_SRC_DIR)/X-ScopedMemoryAccess-bin.java.template > 34: DEST := $(SCOPED_MEMORY_ACCESS_GENSRC_DIR)/ScopedMemoryAccess.java `DEST` is a very generic and not really informative name. Maybe `SCOPED_MEMORY_ACCESS_GENSRC_DEST` to fit in with the rest of the names? And/or, maybe, to cut down on the excessive length, shorten `SCOPED_MEMORY_ACCESS` to `SMA` in all variables. make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 26: > 24: # > 25: > 26: GENSRC_SCOPED_MEMORY_ACCESS := This variable does not seem to be used. A left-over from previous iterations? Also, please cut down a bit on the consecutive empty lines. ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Fri Oct 23 14:09:39 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Fri, 23 Oct 2020 14:09:39 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v12] In-Reply-To: References: <2IKx6cpc-IGP3jZtr0s2I14BWM6ptyFD26szPl3b1ng=.9d956b98-dfe6-4a45-a371-bf86923214fb@github.com> Message-ID: On Fri, 23 Oct 2020 11:02:11 GMT, Magnus Ihse Bursie wrote: >> Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: >> >> Fix whitespaces > > Changes requested by ihse (Reviewer). @magicus the files you commented on are not part of this PR, but they are introduced as part of: https://git.openjdk.java.net/jdk/pull/548 (you seemed to have approved the changes there - but it's also likely that this PR doesn't include the latest changes in that PR). Sorry for the confusion - but please do report any comment you have on the build changes on that PR! ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From sophokles.kh at gmail.com Fri Oct 23 14:56:11 2020 From: sophokles.kh at gmail.com (Kai) Date: Fri, 23 Oct 2020 16:56:11 +0200 Subject: NPE in PKIXCertPathValidator Message-ID: Hi, I ran into a NPE while validating a certificate chain with the latest JDK 11 using a TrustAnchor that has been created using the TrustAnchor(caName, publicKey, nameConstraints) constructor. I suspect the PKIXCertPathValidator.validate(TrustAnchor, ValidatorParams) method to cause the NPE ( http://hg.openjdk.java.net/jdk/jdk/file/ee1d592a9f53/src/java.base/share/classes/sun/security/provider/certpath/PKIXCertPathValidator.java ): X509ValidationEvent xve = new X509ValidationEvent();if (xve.shouldCommit() || EventHelper.isLoggingSecurity()) { int[] certIds = params.certificates().stream() .mapToInt(x -> x.hashCode()) .toArray(); int anchorCertId = anchor.getTrustedCert().hashCode(); if (xve.shouldCommit()) { xve.certificateId = anchorCertId; int certificatePos = 1; //anchor cert xve.certificatePosition = certificatePos; xve.validationCounter = validationCounter.incrementAndGet(); xve.commit(); // now, iterate through remaining for (int id : certIds) { xve.certificateId = id; xve.certificatePosition = ++certificatePos; xve.commit(); } } if (EventHelper.isLoggingSecurity()) { EventHelper.logX509ValidationEvent(anchorCertId, certIds); } } IMHO line int anchorCertId = anchor.getTrustedCert().hashCode(); will throw the NPE if the trust anchor has not been created with a certificate as in my case. The code should do a null check here and fall back to using the hashCode of the PublicKey. WDYT? Kai -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.mullan at oracle.com Fri Oct 23 15:00:42 2020 From: sean.mullan at oracle.com (Sean Mullan) Date: Fri, 23 Oct 2020 11:00:42 -0400 Subject: NPE in PKIXCertPathValidator In-Reply-To: References: Message-ID: <7ca1cbcf-4ee1-4984-df31-1ffa7e4c1650@oracle.com> Yes, that is a bug. Do you want to file a bug report or would you like us to file on one your behalf? Thanks, Sean On 10/23/20 10:56 AM, Kai wrote: > Hi, > > I ran into a NPE while validating a certificate chain with the latest > JDK 11 using a TrustAnchor that has been created using the > TrustAnchor(caName, publicKey, nameConstraints) constructor. > > I suspect the PKIXCertPathValidator.validate(TrustAnchor, > ValidatorParams) method to cause the NPE > (http://hg.openjdk.java.net/jdk/jdk/file/ee1d592a9f53/src/java.base/share/classes/sun/security/provider/certpath/PKIXCertPathValidator.java): > > X509ValidationEvent xve = new X509ValidationEvent(); > if (xve.shouldCommit() || EventHelper.isLoggingSecurity()) { > int[] certIds = params.certificates().stream() > .mapToInt(x -> x.hashCode()) > .toArray(); > int anchorCertId = anchor.getTrustedCert().hashCode(); > if (xve.shouldCommit()) { > xve.certificateId = anchorCertId; > int certificatePos = 1; //anchor cert > xve.certificatePosition = certificatePos; > xve.validationCounter = validationCounter.incrementAndGet(); > xve.commit(); > // now, iterate through remaining > for (int id : certIds) { > xve.certificateId = id; > xve.certificatePosition = ++certificatePos; > xve.commit(); > } > } > if (EventHelper.isLoggingSecurity()) { > EventHelper.logX509ValidationEvent(anchorCertId, certIds); > } > } > > IMHO line > int anchorCertId = anchor.getTrustedCert().hashCode(); > will throw the NPE if the trust anchor has not been created with a > certificate as in my case. > The code should do a null check here and fall back to using the > hashCode of the PublicKey. > WDYT? > > Kai -------------- next part -------------- An HTML attachment was scrubbed... URL: From xuelei.fan at oracle.com Fri Oct 23 15:47:47 2020 From: xuelei.fan at oracle.com (Xuelei Fan) Date: Fri, 23 Oct 2020 08:47:47 -0700 Subject: Request for comment, a new idea about distributed TLS sessions In-Reply-To: <06f7e4af-ccea-7539-9b08-fa9813898255@oracle.com> References: <06f7e4af-ccea-7539-9b08-fa9813898255@oracle.com> Message-ID: <28fc3f1c-6d0a-6d13-4dc3-2b79c468d5dd@oracle.com> Hi, The JEP was updated so that it has a better presentation. https://bugs.openjdk.java.net/browse/JDK-8245551 The goals now is described in a higher level, and some of the details are moved to the Description section. Any comments are welcome. Please let me know by end of this month, October 31, 2020. BTW, I will post a new thread about the algorithm used for the session ticket protection and synchronization in the cluster. Thanks, Xuelei On 9/29/2020 9:25 PM, Xuelei Fan wrote: > Hi, > > I was wondering to improve the scalability of the TLS implementation in > JDK.? TLS session resumption is much faster than full handshaking.? It > may be a good to support efficiently distributing and resuming TLS > sessions across clusters of computers, by using stateless TLS session > tickets. > > The following is a list of the goals: > 1. Use session tickets to distribute and resume sessions. > > 2. Implement a protection scheme for session tickets. > > 3. Deprecate or modify Java SE APIs that negatively impact distributed > session resumption. > > 4. Ensure that the session tickets generated and protected in one server > node can be used for session resumption in other nodes in the > distributed system. > > 5. Ensure that the secret keys used to protect the session ticket can be > rotated and synchronized effectively. > > 6. Ensure that a new server node inserted into the distributed system > can be automatically synchronized, thus making it possible to plugin new > server nodes as needed. > > For more details, please refer to the draft JEP. > > ??? https://bugs.openjdk.java.net/browse/JDK-8245551 > > Does it sound like a good idea?? Did you run into scalability problems > for TLS/HTTPS connections?? Any suggestions?? Any comments are welcome. > > Thanks & Regards, > Xuelei From ascarpino at openjdk.java.net Fri Oct 23 16:38:01 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Fri, 23 Oct 2020 16:38:01 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v3] In-Reply-To: References: Message-ID: > 8253821: Improve ByteBuffer performance with GCM Anthony Scarpino has updated the pull request incrementally with six additional commits since the last revision: - style - style & comments - full update - remove old - update - outputsize ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/411/files - new: https://git.openjdk.java.net/jdk/pull/411/files/b29c1ed5..7c54017f Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=411&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=411&range=01-02 Stats: 857 lines in 6 files changed: 635 ins; 133 del; 89 mod Patch: https://git.openjdk.java.net/jdk/pull/411.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/411/head:pull/411 PR: https://git.openjdk.java.net/jdk/pull/411 From ascarpino at openjdk.java.net Fri Oct 23 16:38:02 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Fri, 23 Oct 2020 16:38:02 GMT Subject: RFR: 8253821: Improve ByteBuffer performance with GCM [v3] In-Reply-To: <9IvsxvHGzvQoM756tUBFHLcMyZjvdbevzh7c-I3i6zU=.03c89dcb-a547-4a07-b0db-358fa9b3fe27@github.com> References: <9IvsxvHGzvQoM756tUBFHLcMyZjvdbevzh7c-I3i6zU=.03c89dcb-a547-4a07-b0db-358fa9b3fe27@github.com> Message-ID: On Wed, 7 Oct 2020 19:16:12 GMT, Valerie Peng wrote: >> Anthony Scarpino has updated the pull request incrementally with six additional commits since the last revision: >> >> - style >> - style & comments >> - full update >> - remove old >> - update >> - outputsize > > I will take a look as well. This is an update for all of Valerie's comments and adding a test to verify different buffer types and configurations ------------- PR: https://git.openjdk.java.net/jdk/pull/411 From dfuchs at openjdk.java.net Fri Oct 23 18:33:35 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Fri, 23 Oct 2020 18:33:35 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Fri, 23 Oct 2020 09:14:48 GMT, Daniel Fuchs wrote: >> The changes in src/java.desktop looks fine. > > Changes to `java.logging` and `java.net.http` also look good to me. Hi Sergey, I'll give it some testing and sponsor it next week unless someone else steps up. best regards, -- daniel ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From valerie.peng at oracle.com Sat Oct 24 01:18:56 2020 From: valerie.peng at oracle.com (Valerie Peng) Date: Fri, 23 Oct 2020 18:18:56 -0700 Subject: Please add HMAC keygen to SunPKCS11 In-Reply-To: References: Message-ID: <5f8bdd12-60eb-b57d-c9d3-6ecf8d8a73e4@oracle.com> Hi, Justin, Most callers just wrap the HMAC key bytes into a java SecretKey object, e.g. new SecretKeySpec(keyBytes, "HmacSHA256"), pass that into the HMAC impl from SunPKCS11 provider which will then convert it into a CKK_GENERIC_SECRET key and passing that to underlying PKCS11 library. Maybe for some very specific cases, support CKM_GENERIC_SECRET_KEY_GEN is necessary and I can look into that. For determining the priority on this, would the java SecretKey object address your need? Or is there other reason requiring 3rd party utility? Thanks, Valerie On 10/21/2020 8:44 PM, Justin Cranford wrote: > > Compare SunPKCS11 support for AES vs HMAC > > * AES => keygen is supported, and AES key can be used for encrypt > and decrypt. > * HMAC => keygen is not supported, but HMAC key can be used for MAC. > > This does not make sense. A third-party utility is required for HMAC > keygen, but not for AES keygen. > > Use case: > > * PKCS#11 driver is v2.20. > * This means AES-256-GCM is not available for confidentiality and > integrity, because GCM supported was only added in PKCS#11 v2.40. > * Fallback to AES-256-CBC and HmacSha256 is required for > confidentiality and integrity, respectively. > * Java can trigger AES keygen, but not HMAC keygen. A third-party > utility is required to trigger HMAC keygen before running Java. > > Would it be possible to add the missing GENERIC-SECRET-KEY-GEN > mechanism to SunPKCS11? Notice how that mechanism is missing from the > documented SunPKCS11 algorithms and mechanisms. It is the same in Java > 8 all the way up to 15. > > * https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html#ALG > > > To reproduce and demonstrate the missing HMAC keygen issue, here is a > small Java Maven project. > > * https://github.com/justincranford/pkcs11 > > > The readme shows the commands to initialize the SoftHSM2 token, and > use a third-party OpenSC utility to trigger HMAC keygen. It also shows > how to set the required SoftHSM2 env variable and run the Maven build. > > The Maven build will execute the ITPkcs11.java integration test class. > The tests demonstrate: > > * Successful SunPKCS11 login to SoftHSM2 and list any existing keys > * Successful AES keygen, encrypt, decrypt > * Successful HMAC mac > * Failed HMAC keygen (because SunPKCS11 does not support > GENERIC-SECRET-KEY-GEN mechanism yet) > > Thank you, > > Justin Cranford > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecki at zusammenkunft.net Sat Oct 24 01:39:01 2020 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Sat, 24 Oct 2020 01:39:01 +0000 Subject: Please add HMAC keygen to SunPKCS11 In-Reply-To: <5f8bdd12-60eb-b57d-c9d3-6ecf8d8a73e4@oracle.com> References: , <5f8bdd12-60eb-b57d-c9d3-6ecf8d8a73e4@oracle.com> Message-ID: Hello, I would agree with this request, my usecase would be to use a HSM, where I typically don?t want to import keys but generate them safely on the HSM so not even admins have access to the key material ever (besides maybe having a key handle to wrap it). Isn?t that what the KeyGen interface is all about? Such cases are not tha easy to model with the current abstract PKCS11 Support it seems. Gruss Bernd -- http://bernd.eckenfels.net ________________________________ Von: security-dev im Auftrag von Valerie Peng Gesendet: Saturday, October 24, 2020 3:18:56 AM An: security-dev at openjdk.java.net Betreff: Re: Please add HMAC keygen to SunPKCS11 Hi, Justin, Most callers just wrap the HMAC key bytes into a java SecretKey object, e.g. new SecretKeySpec(keyBytes, "HmacSHA256"), pass that into the HMAC impl from SunPKCS11 provider which will then convert it into a CKK_GENERIC_SECRET key and passing that to underlying PKCS11 library. Maybe for some very specific cases, support CKM_GENERIC_SECRET_KEY_GEN is necessary and I can look into that. For determining the priority on this, would the java SecretKey object address your need? Or is there other reason requiring 3rd party utility? Thanks, Valerie On 10/21/2020 8:44 PM, Justin Cranford wrote: Compare SunPKCS11 support for AES vs HMAC * AES => keygen is supported, and AES key can be used for encrypt and decrypt. * HMAC => keygen is not supported, but HMAC key can be used for MAC. This does not make sense. A third-party utility is required for HMAC keygen, but not for AES keygen. Use case: * PKCS#11 driver is v2.20. * This means AES-256-GCM is not available for confidentiality and integrity, because GCM supported was only added in PKCS#11 v2.40. * Fallback to AES-256-CBC and HmacSha256 is required for confidentiality and integrity, respectively. * Java can trigger AES keygen, but not HMAC keygen. A third-party utility is required to trigger HMAC keygen before running Java. Would it be possible to add the missing GENERIC-SECRET-KEY-GEN mechanism to SunPKCS11? Notice how that mechanism is missing from the documented SunPKCS11 algorithms and mechanisms. It is the same in Java 8 all the way up to 15. * https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html#ALG To reproduce and demonstrate the missing HMAC keygen issue, here is a small Java Maven project. * https://github.com/justincranford/pkcs11 The readme shows the commands to initialize the SoftHSM2 token, and use a third-party OpenSC utility to trigger HMAC keygen. It also shows how to set the required SoftHSM2 env variable and run the Maven build. The Maven build will execute the ITPkcs11.java integration test class. The tests demonstrate: * Successful SunPKCS11 login to SoftHSM2 and list any existing keys * Successful AES keygen, encrypt, decrypt * Successful HMAC mac * Failed HMAC keygen (because SunPKCS11 does not support GENERIC-SECRET-KEY-GEN mechanism yet) Thank you, Justin Cranford -------------- next part -------------- An HTML attachment was scrubbed... URL: From prr at openjdk.java.net Sat Oct 24 23:14:34 2020 From: prr at openjdk.java.net (Phil Race) Date: Sat, 24 Oct 2020 23:14:34 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 20:46:15 GMT, ?????? ??????? wrote: > As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: > @State(Scope.Thread) > @OutputTimeUnit(TimeUnit.NANOSECONDS) > @BenchmarkMode(value = Mode.AverageTime) > public class AtomicBenchmark { > @Benchmark > public Object defaultValue() { > return new AtomicInteger(); > } > @Benchmark > public Object explicitValue() { > return new AtomicInteger(0); > } > } > THis benchmark demonstrates that `explicitValue()` is much slower: > Benchmark Mode Cnt Score Error Units > AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op > AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op > So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. > > I've tested the changes locally, both tier1 and tier 2 are ok. > > Could one create an issue for tracking this? client changes are fine ------------- Marked as reviewed by prr (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/818 From Justin.Cranford at entrust.com Mon Oct 26 02:59:03 2020 From: Justin.Cranford at entrust.com (Justin Cranford) Date: Mon, 26 Oct 2020 02:59:03 +0000 Subject: Please add HMAC keygen to SunPKCS11 Message-ID: Hi Valerie, Thank you for responding to my request for HMAC keygen in SunPKCS11. The workaround you mentioned executes OK, but does not address the use case. SunJCE handles the same key object for HmacSHA256 too, so I am not sure if there is any benefit to passing a key into SunPKCS11. Here is the use case. I would like to generate all keys inside the HSM and have them marked unexportable. That strongly guarantees keys never leave the HSM. It also requires continuous access to the HSM which can be managed (ex: physical or network access). Finally, it requires authentication to use the keys which can also be managed (i.e. user PIN with reset/update by SO user). The only reason for me to use a 3rd party PKCS#11 client is requesting generic secret keygen inside the HSM. After that, SunPKCS11 can login and use it. If SunPKCS11 can handle requesting generic secret keygen, then I don't need that 3rd party HSM client anymore. Thank you, Justin Cranford ------------------------------ Date: Fri, 23 Oct 2020 18:18:56 -0700 From: Valerie Peng To: security-dev at openjdk.java.net Subject: Re: Please add HMAC keygen to SunPKCS11 Message-ID: <5f8bdd12-60eb-b57d-c9d3-6ecf8d8a73e4 at oracle.com> Content-Type: text/plain; charset="windows-1252"; Format="flowed" Hi, Justin, Most callers just wrap the HMAC key bytes into a java SecretKey object, e.g. new SecretKeySpec(keyBytes, "HmacSHA256"), pass that into the HMAC impl from SunPKCS11 provider which will then convert it into a CKK_GENERIC_SECRET key and passing that to underlying PKCS11 library. Maybe for some very specific cases, support CKM_GENERIC_SECRET_KEY_GEN is necessary and I can look into that. For determining the priority on this, would the java SecretKey object address your need? Or is there other reason requiring 3rd party utility? Thanks, Valerie From ihse at openjdk.java.net Mon Oct 26 09:14:41 2020 From: ihse at openjdk.java.net (Magnus Ihse Bursie) Date: Mon, 26 Oct 2020 09:14:41 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v13] In-Reply-To: References: Message-ID: On Mon, 19 Oct 2020 10:34:32 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. >> >> This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). >> >> A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. >> >> A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. >> >> Thanks >> Maurizio >> >> Javadoc: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. >> >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. >> >> After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. >> >> Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). >> >> The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. >> >> As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. >> >> In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. >> >> To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). >> >> Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). >> >> `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. >> >> The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. >> >> #### Memory access var handles overhaul >> >> The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. >> >> This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. >> >> This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. >> >> #### Test changes >> >> Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. >> >> [1] - https://openjdk.java.net/jeps/393 >> [2] - https://openjdk.java.net/jeps/389 >> [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html >> [4] - https://openjdk.java.net/jeps/312 > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Address CSR comments Changes requested by ihse (Reviewer). make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 148: > 146: > 147: $(DEST): $(BUILD_TOOLS_JDK) $(SCOPED_MEMORY_ACCESS_TEMPLATE) $(SCOPED_MEMORY_ACCESS_BIN_TEMPLATE) > 148: $(MKDIR) -p $(SCOPED_MEMORY_ACCESS_GENSRC_DIR) Please use `$(call MakeDir, $(SCOPED_MEMORY_ACCESS_GENSRC_DIR))` instead. make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 34: > 32: SCOPED_MEMORY_ACCESS_TEMPLATE := $(SCOPED_MEMORY_ACCESS_SRC_DIR)/X-ScopedMemoryAccess.java.template > 33: SCOPED_MEMORY_ACCESS_BIN_TEMPLATE := $(SCOPED_MEMORY_ACCESS_SRC_DIR)/X-ScopedMemoryAccess-bin.java.template > 34: DEST := $(SCOPED_MEMORY_ACCESS_GENSRC_DIR)/ScopedMemoryAccess.java `DEST` is a very generic and not really informative name. Maybe `SCOPED_MEMORY_ACCESS_GENSRC_DEST` to fit in with the rest of the names? And/or, maybe, to cut down on the excessive length, shorten `SCOPED_MEMORY_ACCESS` to `SMA` in all variables. make/modules/java.base/gensrc/GensrcScopedMemoryAccess.gmk line 26: > 24: # > 25: > 26: GENSRC_SCOPED_MEMORY_ACCESS := This variable does not seem to be used. A left-over from previous iterations? Also, please cut down a bit on the consecutive empty lines. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From ihse at openjdk.java.net Mon Oct 26 09:19:37 2020 From: ihse at openjdk.java.net (Magnus Ihse Bursie) Date: Mon, 26 Oct 2020 09:19:37 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v12] In-Reply-To: <2IKx6cpc-IGP3jZtr0s2I14BWM6ptyFD26szPl3b1ng=.9d956b98-dfe6-4a45-a371-bf86923214fb@github.com> References: <2IKx6cpc-IGP3jZtr0s2I14BWM6ptyFD26szPl3b1ng=.9d956b98-dfe6-4a45-a371-bf86923214fb@github.com> Message-ID: On Thu, 22 Oct 2020 17:04:34 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the first incubation round of the foreign linker access API incubation >> (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). >> >> The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. >> >> Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. >> >> A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). >> >> Thanks >> Maurizio >> >> Webrev: >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev >> >> Javadoc: >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html >> >> Specdiff (relative to [3]): >> >> http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254232 >> >> >> >> ### API Changes >> >> The API changes are actually rather slim: >> >> * `LibraryLookup` >> * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. >> * `FunctionDescriptor` >> * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. >> * `CLinker` >> * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. >> * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. >> * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. >> * `NativeScope` >> * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. >> * `MemorySegment` >> * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. >> >> ### Safety >> >> The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). >> >> ### Implementation changes >> >> The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). >> >> As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. >> >> Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. >> >> The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. >> >> This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. >> >> For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. >> >> A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. >> >> At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: >> >> * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. >> >> * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). >> >> * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. >> >> For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). >> >> Again, for more readings on the internals of the foreign linker support, please refer to [5]. >> >> #### Test changes >> >> Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. >> >> Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. >> >> [1] - https://openjdk.java.net/jeps/389 >> [2] - https://openjdk.java.net/jeps/393 >> [3] - https://git.openjdk.java.net/jdk/pull/548 >> [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md >> [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Fix whitespaces Marked as reviewed by ihse (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From ihse at openjdk.java.net Mon Oct 26 09:19:37 2020 From: ihse at openjdk.java.net (Magnus Ihse Bursie) Date: Mon, 26 Oct 2020 09:19:37 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v12] In-Reply-To: References: <2IKx6cpc-IGP3jZtr0s2I14BWM6ptyFD26szPl3b1ng=.9d956b98-dfe6-4a45-a371-bf86923214fb@github.com> Message-ID: On Fri, 23 Oct 2020 14:06:22 GMT, Maurizio Cimadamore wrote: >> Changes requested by ihse (Reviewer). > > @magicus the files you commented on are not part of this PR, but they are introduced as part of: > https://git.openjdk.java.net/jdk/pull/548 > (you seemed to have approved the changes there - but it's also likely that this PR doesn't include the latest changes in that PR). Sorry for the confusion - but please do report any comment you have on the build changes on that PR! @mcimadamore I'm sorry too for the confusion. :) I must have been a bit in a bit of a hurry when approving it on the other PR. I've now moved my comments there. I don't think there's any way for me to "un-review" this change, so I'll mark it as accepted, even though I don't have anything to say about it (so that I'm not blocking a push). I'll ask the Skara guys if there's a better way to deal with this. Also, in the future, if you are creating a PR which Skara believes has changes in the build system, but it "really" does not, please remove the `build` label, and I won't even see the PR to come bothering you again! ;-) ------------- PR: https://git.openjdk.java.net/jdk/pull/634 From magnus.ihse.bursie at oracle.com Mon Oct 26 12:31:06 2020 From: magnus.ihse.bursie at oracle.com (Magnus Ihse Bursie) Date: Mon, 26 Oct 2020 13:31:06 +0100 Subject: RFR: 8235710: Remove the legacy elliptic curves [v2] In-Reply-To: References: Message-ID: <261dec99-31d0-d8d1-f73d-cc4378cc2008@oracle.com> Sorry for being late on this one (I'm working through a huge backlog), but it does not seem like the removal was complete. ENABLE_INTREE_EC is still present in spec.gmk. And it is still checked in modules/jdk.crypto.ec/Lib.gmk. In fact, this entire file should be removed. Anthony, can you please open a new JBS issue to fix the remaining cleanup? /Magnus On 2020-09-22 15:23, Erik Joelsson wrote: > On Tue, 22 Sep 2020 00:18:07 GMT, Anthony Scarpino wrote: > >>> This change removes the native elliptic curves library code; as well as, and calls to that code, tests, and files >>> associated with those libraries. The makefiles have been changed to remove from all source builds of the ec code. The >>> SunEC system property is removed and java.security configurations changed to reflect the removed curves. This will >>> remove the following elliptic curves from SunEC: secp112r1, secp112r2, secp128r1, secp128r2, secp160k1, secp160r1, >>> secp160r2, secp192k1, secp192r1, secp224k1, secp224r1, secp256k1, sect113r1, sect113r2, sect131r1, sect131r2, >>> sect163k1, sect163r1, sect163r2, sect193r1, sect193r2, sect233k1, sect233r1, sect239k1, sect283k1, sect283r1, >>> sect409k1, sect409r1, sect571k1, sect571r1, X9.62 c2tnb191v1, X9.62 c2tnb191v2, X9.62 c2tnb191v3, X9.62 c2tnb239v1, >>> X9.62 c2tnb239v2, X9.62 c2tnb239v3, X9.62 c2tnb359v1, X9.62 c2tnb431r1, X9.62 prime192v2, X9.62 prime192v3, X9.62 >>> prime239v1, X9.62 prime239v2, X9.62 prime239v3, brainpoolP256r1 brainpoolP320r1, brainpoolP384r1, brainpoolP512r1 >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> remove JDKOPT_DETECT_INTREE_EC from configure.ac > Build changes look good. > > Marked as reviewed by erikj (Reviewer). > > ------------- > > PR: https://git.openjdk.java.net/jdk/pull/289 From weijun at openjdk.java.net Mon Oct 26 14:13:16 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Mon, 26 Oct 2020 14:13:16 GMT Subject: RFR: 8255393: sun/security/util/DerValue/Indefinite.java fails with ---illegal-access=deny Message-ID: The test calls Field::setAccessible and needs an "open" access to the internal class. ------------- Commit messages: - 8255393: sun/security/util/DerValue/Indefinite.java fails with ---illegal-access=deny Changes: https://git.openjdk.java.net/jdk/pull/864/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=864&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8255393 Stats: 3 lines in 1 file changed: 0 ins; 1 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/864.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/864/head:pull/864 PR: https://git.openjdk.java.net/jdk/pull/864 From weijun at openjdk.java.net Mon Oct 26 14:38:26 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Mon, 26 Oct 2020 14:38:26 GMT Subject: RFR: 8255393: sun/security/util/DerValue/Indefinite.java fails with ---illegal-access=deny [v2] In-Reply-To: References: Message-ID: > The test calls Field::setAccessible and needs an "open" access to the internal class. Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: no new bug id with ---illegal-access=deny ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/864/files - new: https://git.openjdk.java.net/jdk/pull/864/files/dac4c654..40b30911 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=864&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=864&range=00-01 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/864.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/864/head:pull/864 PR: https://git.openjdk.java.net/jdk/pull/864 From alanb at openjdk.java.net Mon Oct 26 14:38:27 2020 From: alanb at openjdk.java.net (Alan Bateman) Date: Mon, 26 Oct 2020 14:38:27 GMT Subject: RFR: 8255393: sun/security/util/DerValue/Indefinite.java fails with ---illegal-access=deny [v2] In-Reply-To: References: Message-ID: On Mon, 26 Oct 2020 14:35:23 GMT, Weijun Wang wrote: >> The test calls Field::setAccessible and needs an "open" access to the internal class. > > Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: > > no new bug id with ---illegal-access=deny Marked as reviewed by alanb (Reviewer). test/jdk/sun/security/util/DerValue/Indefinite.java line 26: > 24: /* > 25: * @test > 26: * @bug 6731685 8249783 8255393 I don't think you need to extend the bug tag for this issue as it doesn't add/change the test. ------------- PR: https://git.openjdk.java.net/jdk/pull/864 From weijun at openjdk.java.net Mon Oct 26 14:38:28 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Mon, 26 Oct 2020 14:38:28 GMT Subject: Integrated: 8255393: sun/security/util/DerValue/Indefinite.java fails with ---illegal-access=deny In-Reply-To: References: Message-ID: On Mon, 26 Oct 2020 14:07:36 GMT, Weijun Wang wrote: > The test calls Field::setAccessible and needs an "open" access to the internal class. This pull request has now been integrated. Changeset: e8b75b13 Author: Weijun Wang URL: https://git.openjdk.java.net/jdk/commit/e8b75b13 Stats: 2 lines in 1 file changed: 0 ins; 1 del; 1 mod 8255393: sun/security/util/DerValue/Indefinite.java fails with ---illegal-access=deny Reviewed-by: alanb ------------- PR: https://git.openjdk.java.net/jdk/pull/864 From valerie.peng at oracle.com Mon Oct 26 16:58:55 2020 From: valerie.peng at oracle.com (Valerie Peng) Date: Mon, 26 Oct 2020 09:58:55 -0700 Subject: Please add HMAC keygen to SunPKCS11 In-Reply-To: References: <5f8bdd12-60eb-b57d-c9d3-6ecf8d8a73e4@oracle.com> Message-ID: <8f95356e-18bd-9591-7fc4-abe25c65de75@oracle.com> True, using HSM is one scenario that would require the key material be directly from the underlying PKCS11 library. I've filed a RFE about this and marked it P3. https://bugs.openjdk.java.net/browse/JDK-8255407 Thanks for the input. Valerie On 10/23/2020 6:39 PM, Bernd Eckenfels wrote: > Hello, > > I would agree with this request, my usecase would be to use a HSM, > where I typically don?t want to import keys but generate them safely > on the HSM so not even admins have access to the key ?material ever > (besides maybe having a key handle to wrap it). Isn?t that what the > KeyGen interface is all about? > > Such cases are not tha easy to model with the current abstract PKCS11 > Support it seems. > > Gruss > Bernd > -- > http://bernd.eckenfels.net > ------------------------------------------------------------------------ > *Von:* security-dev im Auftrag > von Valerie Peng > *Gesendet:* Saturday, October 24, 2020 3:18:56 AM > *An:* security-dev at openjdk.java.net > *Betreff:* Re: Please add HMAC keygen to SunPKCS11 > > Hi, Justin, > > Most callers just wrap the HMAC key bytes into a java SecretKey > object, e.g. new SecretKeySpec(keyBytes, "HmacSHA256"), pass that into > the HMAC impl from SunPKCS11 provider which will then convert it into > a CKK_GENERIC_SECRET key and passing that to underlying PKCS11 library. > > Maybe for some very specific cases, support CKM_GENERIC_SECRET_KEY_GEN > is necessary and I can look into that. For determining the priority on > this, would the java SecretKey object address your need? Or is there > other reason requiring 3rd party utility? > > Thanks, > Valerie > > > On 10/21/2020 8:44 PM, Justin Cranford wrote: >> >> Compare SunPKCS11 support for AES vs HMAC >> >> * AES => keygen is supported, and AES key can be used for encrypt >> and decrypt. >> * HMAC => keygen is not supported, but HMAC key can be used for MAC. >> >> This does not make sense. A third-party utility is required for HMAC >> keygen, but not for AES keygen. >> >> Use case: >> >> * PKCS#11 driver is v2.20. >> * This means AES-256-GCM is not available for confidentiality and >> integrity, because GCM supported was only added in PKCS#11 v2.40. >> * Fallback to AES-256-CBC and HmacSha256 is required for >> confidentiality and integrity, respectively. >> * Java can trigger AES keygen, but not HMAC keygen. A third-party >> utility is required to trigger HMAC keygen before running Java. >> >> Would it be possible to add the missing GENERIC-SECRET-KEY-GEN >> mechanism to SunPKCS11? Notice how that mechanism is missing from the >> documented SunPKCS11 algorithms and mechanisms. It is the same in >> Java 8 all the way up to 15. >> >> * https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html#ALG >> >> >> To reproduce and demonstrate the missing HMAC keygen issue, here is a >> small Java Maven project. >> >> * https://github.com/justincranford/pkcs11 >> >> >> The readme shows the commands to initialize the SoftHSM2 token, and >> use a third-party OpenSC utility to trigger HMAC keygen. It also >> shows how to set the required SoftHSM2 env variable and run the Maven >> build. >> >> The Maven build will execute the ITPkcs11.java integration test >> class. The tests demonstrate: >> >> * Successful SunPKCS11 login to SoftHSM2 and list any existing keys >> * Successful AES keygen, encrypt, decrypt >> * Successful HMAC mac >> * Failed HMAC keygen (because SunPKCS11 does not support >> GENERIC-SECRET-KEY-GEN mechanism yet) >> >> Thank you, >> >> Justin Cranford >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From abakhtin at openjdk.java.net Tue Oct 27 12:35:25 2020 From: abakhtin at openjdk.java.net (Alexey Bakhtin) Date: Tue, 27 Oct 2020 12:35:25 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore Message-ID: Hi All, DES and DESede keys are supported by JKS/JCEKS but not supported by PKCS#12 keystores. This issue prevents the migration of legacy applications to PKCS#12 keystore. For example, an application has some old 3DES keys that are required for certain legacy features. Java PKCS12 keystore does not support DES/3DES keys, thus, application can?t migrate to PKCS#12 This patch adds OIDs for the DES/DESede algorithms. It is the only changes required to support DES/3DES keys in the PKCS#12 keystore. sun/security/pkcs12/P12SecretKey test is updated to verify new secret keys in the PKCS#12 keystore. ------------- Commit messages: - 8007632: DES/3DES keys support in PKCS12 keystore Changes: https://git.openjdk.java.net/jdk/pull/877/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=877&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8007632 Stats: 16 lines in 2 files changed: 6 ins; 1 del; 9 mod Patch: https://git.openjdk.java.net/jdk/pull/877.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/877/head:pull/877 PR: https://git.openjdk.java.net/jdk/pull/877 From mcimadamore at openjdk.java.net Tue Oct 27 12:59:31 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 27 Oct 2020 12:59:31 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v14] In-Reply-To: References: Message-ID: <6DXvMCLE3m-Rcv_Sxl-WMZITXkY8LwzctCgf_Ke83Ls=.c8f71977-0f81-415f-b6a5-f56be3a934c1@github.com> > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. > > This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). > > A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. > > A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. > > Thanks > Maurizio > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. > > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. > > After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. > > Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). > > The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. > > As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. > > In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. > > To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). > > Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). > > `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. > > The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. > > #### Memory access var handles overhaul > > The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. > > This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. > > This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. > > #### Test changes > > Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. > > [1] - https://openjdk.java.net/jeps/393 > [2] - https://openjdk.java.net/jeps/389 > [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html > [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 18 commits: - Merge branch 'master' into 8254162 - Address review comment for scoped memory access makefile - Address CSR comments - Back-port of TestByteBuffer fix - Merge branch 'master' into 8254162 - Merge branch 'master' into 8254162 - Merge branch 'master' into 8254162 - Remove spurious check on MemoryScope::confineTo Added tests to make sure no spurious exception is thrown when: * handing off a segment from A to A * sharing an already shared segment - Merge branch 'master' into 8254162 - Simplify example in the toplevel javadoc - ... and 8 more: https://git.openjdk.java.net/jdk/compare/cf56c7e0...697c7ca5 ------------- Changes: https://git.openjdk.java.net/jdk/pull/548/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=13 Stats: 7504 lines in 79 files changed: 4797 ins; 1530 del; 1177 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From weijun at openjdk.java.net Tue Oct 27 14:21:20 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 27 Oct 2020 14:21:20 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore In-Reply-To: References: Message-ID: <_SPAmdaNfo0W8wFmFt3kkregKBQcMnW4UDQIaxYZAJU=.2304ad48-b3b7-4de3-95ac-7c49c3967fcb@github.com> On Tue, 27 Oct 2020 12:27:52 GMT, Alexey Bakhtin wrote: > Hi All, > > DES and DESede keys are supported by JKS/JCEKS but not supported by PKCS#12 keystores. > This issue prevents the migration of legacy applications to PKCS#12 keystore. For example, an application has some old 3DES keys that are required for certain legacy features. Java PKCS12 keystore does not support DES/3DES keys, thus, application can?t migrate to PKCS#12 > This patch adds OIDs for the DES/DESede algorithms. It is the only changes required to support DES/3DES keys in the PKCS#12 keystore. > sun/security/pkcs12/P12SecretKey test is updated to verify new secret keys in the PKCS#12 keystore. src/java.base/share/classes/sun/security/util/KnownOIDs.java line 355: > 353: // OIW secsig 1.3.14.3.* > 354: OIW_DES_CBC("1.3.14.3.2.7", "DES/CBC"), > 355: OIW_DES_ECB("1.3.14.3.2.6", "DES/ECB", "DES"), What OID are other vendors using? ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From shade at openjdk.java.net Tue Oct 27 14:36:25 2020 From: shade at openjdk.java.net (Aleksey Shipilev) Date: Tue, 27 Oct 2020 14:36:25 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) In-Reply-To: References: Message-ID: On Wed, 7 Oct 2020 17:13:22 GMT, Maurizio Cimadamore wrote: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. > > This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). > > A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. > > A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. > > Thanks > Maurizio > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. > > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. > > After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. > > Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). > > The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. > > As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. > > In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. > > To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). > > Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). > > `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. > > The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. > > #### Memory access var handles overhaul > > The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. > > This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. > > This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. > > #### Test changes > > Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. > > [1] - https://openjdk.java.net/jeps/393 > [2] - https://openjdk.java.net/jeps/389 > [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html > [4] - https://openjdk.java.net/jeps/312 @mcimadamore, if you pull from current master, you would get the Linux x86_32 tier1 run "for free". ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Tue Oct 27 14:43:33 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 27 Oct 2020 14:43:33 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v15] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. > > This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). > > A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. > > A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. > > Thanks > Maurizio > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. > > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. > > After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. > > Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). > > The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. > > As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. > > In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. > > To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). > > Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). > > `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. > > The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. > > #### Memory access var handles overhaul > > The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. > > This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. > > This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. > > #### Test changes > > Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. > > [1] - https://openjdk.java.net/jeps/393 > [2] - https://openjdk.java.net/jeps/389 > [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html > [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 21 commits: - Remove TestMismatch from 32-bit problem list - Merge branch 'master' into 8254162 - Tweak javadoc for MemorySegment::mapFromPath Tweak alignment for long/double Java layouts on 32 bits platforms - Merge branch 'master' into 8254162 - Address review comment for scoped memory access makefile - Address CSR comments - Back-port of TestByteBuffer fix - Merge branch 'master' into 8254162 - Merge branch 'master' into 8254162 - Merge branch 'master' into 8254162 - ... and 11 more: https://git.openjdk.java.net/jdk/compare/7d41a541...f844f544 ------------- Changes: https://git.openjdk.java.net/jdk/pull/548/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=14 Stats: 7526 lines in 80 files changed: 4814 ins; 1531 del; 1181 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Tue Oct 27 14:43:33 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 27 Oct 2020 14:43:33 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) In-Reply-To: References: Message-ID: On Wed, 7 Oct 2020 17:13:22 GMT, Maurizio Cimadamore wrote: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. > > This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). > > A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. > > A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. > > Thanks > Maurizio > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. > > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. > > After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. > > Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). > > The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. > > As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. > > In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. > > To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). > > Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). > > `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. > > The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. > > #### Memory access var handles overhaul > > The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. > > This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. > > This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. > > #### Test changes > > Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. > > [1] - https://openjdk.java.net/jeps/393 > [2] - https://openjdk.java.net/jeps/389 > [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html > [4] - https://openjdk.java.net/jeps/312 > @mcimadamore, if you pull from current master, you would get the Linux x86_32 tier1 run "for free". Just did that - I also removed TestMismatch from the problem list in the latest iteration, and fixed the alignment for long/double layouts, after chatting with the team (https://bugs.openjdk.java.net/browse/JDK-8255350) ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From abakhtin at openjdk.java.net Tue Oct 27 14:47:25 2020 From: abakhtin at openjdk.java.net (Alexey Bakhtin) Date: Tue, 27 Oct 2020 14:47:25 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore In-Reply-To: <_SPAmdaNfo0W8wFmFt3kkregKBQcMnW4UDQIaxYZAJU=.2304ad48-b3b7-4de3-95ac-7c49c3967fcb@github.com> References: <_SPAmdaNfo0W8wFmFt3kkregKBQcMnW4UDQIaxYZAJU=.2304ad48-b3b7-4de3-95ac-7c49c3967fcb@github.com> Message-ID: On Tue, 27 Oct 2020 14:19:00 GMT, Weijun Wang wrote: >> Hi All, >> >> DES and DESede keys are supported by JKS/JCEKS but not supported by PKCS#12 keystores. >> This issue prevents the migration of legacy applications to PKCS#12 keystore. For example, an application has some old 3DES keys that are required for certain legacy features. Java PKCS12 keystore does not support DES/3DES keys, thus, application can?t migrate to PKCS#12 >> This patch adds OIDs for the DES/DESede algorithms. It is the only changes required to support DES/3DES keys in the PKCS#12 keystore. >> sun/security/pkcs12/P12SecretKey test is updated to verify new secret keys in the PKCS#12 keystore. > > src/java.base/share/classes/sun/security/util/KnownOIDs.java line 355: > >> 353: // OIW secsig 1.3.14.3.* >> 354: OIW_DES_CBC("1.3.14.3.2.7", "DES/CBC"), >> 355: OIW_DES_ECB("1.3.14.3.2.6", "DES/ECB", "DES"), > > What OID are other vendors using? At least BouncyCastle uses the same OIDs: http://javadox.com/org.bouncycastle/bcprov-jdk15on/1.51/org/bouncycastle/asn1/oiw/OIWObjectIdentifiers.html ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From mcimadamore at openjdk.java.net Tue Oct 27 15:53:31 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 27 Oct 2020 15:53:31 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v16] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. > > This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). > > A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. > > A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. > > Thanks > Maurizio > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. > > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. > > After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. > > Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). > > The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. > > As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. > > In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. > > To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). > > Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). > > `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. > > The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. > > #### Memory access var handles overhaul > > The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. > > This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. > > This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. > > #### Test changes > > Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. > > [1] - https://openjdk.java.net/jeps/393 > [2] - https://openjdk.java.net/jeps/389 > [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html > [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: * Add final to MappedByteBuffer::SCOPED_MEMORY_ACCESS field * Tweak TestLayouts to make it 32-bit friendly after recent MemoryLayouts tweaks ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/f844f544..e43f5d76 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=15 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=14-15 Stats: 8 lines in 2 files changed: 0 ins; 3 del; 5 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From weijun at openjdk.java.net Tue Oct 27 16:15:23 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 27 Oct 2020 16:15:23 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore In-Reply-To: References: <_SPAmdaNfo0W8wFmFt3kkregKBQcMnW4UDQIaxYZAJU=.2304ad48-b3b7-4de3-95ac-7c49c3967fcb@github.com> Message-ID: On Tue, 27 Oct 2020 14:44:09 GMT, Alexey Bakhtin wrote: >> src/java.base/share/classes/sun/security/util/KnownOIDs.java line 355: >> >>> 353: // OIW secsig 1.3.14.3.* >>> 354: OIW_DES_CBC("1.3.14.3.2.7", "DES/CBC"), >>> 355: OIW_DES_ECB("1.3.14.3.2.6", "DES/ECB", "DES"), >> >> What OID are other vendors using? > > At least BouncyCastle uses the same OIDs: http://javadox.com/org.bouncycastle/bcprov-jdk15on/1.51/org/bouncycastle/asn1/oiw/OIWObjectIdentifiers.html Yes I can see the OIDs. I was just wondering if other vendors are also using the OID for DES/ECB when storing a DES key. After all, this is only a key and it can can be used with all Cipher modes. ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From abakhtin at openjdk.java.net Tue Oct 27 16:56:26 2020 From: abakhtin at openjdk.java.net (Alexey Bakhtin) Date: Tue, 27 Oct 2020 16:56:26 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore In-Reply-To: References: <_SPAmdaNfo0W8wFmFt3kkregKBQcMnW4UDQIaxYZAJU=.2304ad48-b3b7-4de3-95ac-7c49c3967fcb@github.com> Message-ID: <_13F4nXTLb0ew1eDmeiSj04MMk6ScGFa_heNHittcpk=.ff70163d-8c2a-4466-b069-d4145b63f994@github.com> On Tue, 27 Oct 2020 16:12:14 GMT, Weijun Wang wrote: >> At least BouncyCastle uses the same OIDs: http://javadox.com/org.bouncycastle/bcprov-jdk15on/1.51/org/bouncycastle/asn1/oiw/OIWObjectIdentifiers.html > > Yes I can see the OIDs. I was just wondering if other vendors are also using the OID for DES/ECB when storing a DES key. After all, this is only a key and it can can be used with all Cipher modes. Verified with BC: BouncyCastle uses DES/CBC OID for DES secret key in PKCS#12. So, the patch for "DES" can be simplified to - OIW_DES_CBC("1.3.14.3.2.7", "DES/CBC"), + OIW_DES_CBC("1.3.14.3.2.7", "DES/CBC", "DES"), ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From weijun at openjdk.java.net Tue Oct 27 17:33:18 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 27 Oct 2020 17:33:18 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore In-Reply-To: <_13F4nXTLb0ew1eDmeiSj04MMk6ScGFa_heNHittcpk=.ff70163d-8c2a-4466-b069-d4145b63f994@github.com> References: <_SPAmdaNfo0W8wFmFt3kkregKBQcMnW4UDQIaxYZAJU=.2304ad48-b3b7-4de3-95ac-7c49c3967fcb@github.com> <_13F4nXTLb0ew1eDmeiSj04MMk6ScGFa_heNHittcpk=.ff70163d-8c2a-4466-b069-d4145b63f994@github.com> Message-ID: <1-Ge_AsMxv18ZjIumMYUymfLOLMhZObHkpTTSo2OEHc=.606d82a1-d760-4b2a-ba00-c5965d36f333@github.com> On Tue, 27 Oct 2020 16:52:03 GMT, Alexey Bakhtin wrote: >> Yes I can see the OIDs. I was just wondering if other vendors are also using the OID for DES/ECB when storing a DES key. After all, this is only a key and it can can be used with all Cipher modes. > > Verified with BC: BouncyCastle uses DES/CBC OID for DES secret key in PKCS#12. > So, the patch for "DES" can be simplified to > - OIW_DES_CBC("1.3.14.3.2.7", "DES/CBC"), > + OIW_DES_CBC("1.3.14.3.2.7", "DES/CBC", "DES"), Good. I also just noticed that the BC provider uses 1.3.14.3.2.7 as alias for KeyGenerator.DES. ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From djelinski1 at gmail.com Tue Oct 27 17:44:26 2020 From: djelinski1 at gmail.com (=?UTF-8?Q?Daniel_Jeli=C5=84ski?=) Date: Tue, 27 Oct 2020 18:44:26 +0100 Subject: ECC Key Usage ignored Message-ID: Hi all, TL;DR: both SSL server and client ignore KeyUsage certificate extension when determining the list of available cipher suites. They shouldn't; KeyUsage is the only differentiator between ECDH and ECDSA certificates. Long version: I'm experimenting with ECC certificates on my Jetty server; when I created an ECC certificate and tested the server with nmap, I found that both ECDSA and ECDH cipher suites are enabled. I don't want ECDH ciphers, but I don't want to add explicit excludes either. Reading into NIST recommendations [1] I found that ECDSA certificates should define KeyUsage extension with value digitalSignature, vs ECDH which should use keyAgreement value. I experimented with both combinations of KeyValue, both resulted in the same set of ciphers being offered by the server. The client doesn't seem to care about KeyUsage either - it accepts connections even when the selected cipher doesn't match KeyUsage. Chrome browser doesn't support ECDH ciphers, but it does support ECDSA. When connecting to a Java server using ECDH certificate, it displays the error "ERR_SSL_KEY_USAGE_INCOMPATIBLE"; the server offers an ECDSA cipher suite, which is rejected by the browser. The issue was already reported by Bernd Eckenfels here [2], but as far as I can tell, it is not addressed yet; I was able to reproduce it using slightly modified code of this gist [3]. Certificates were generated using keytool commands: ECDSA: keytool -genkeypair -alias ec -keyalg EC -keysize 256 -sigalg SHA256withECDSA -validity 365 -dname "CN=localhost,OU=Unknown,O=Unknown,L=Unknown,S=Unknown,C=Unknown" -storetype JKS -keystore ectest.jks -storepass 123456 -ext KeyUsage:c=digitalSignature,keyCertSign ECDH: keytool -genkeypair -alias ec -keyalg EC -keysize 256 -sigalg SHA256withECDSA -validity 365 -dname "CN=localhost,OU=Unknown,O=Unknown,L=Unknown,S=Unknown,C=Unknown" -storetype JKS -keystore ectest.jks -storepass 123456 -ext KeyUsage:c=keyAgreement,keyCertSign I'm not sure if keyCertSign is required on self-signed certificates, added it just in case. Tested on OpenJDK 11.0.6. Regards, Daniel Jeli?ski [1] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-52r2.pdf [2] http://mail.openjdk.java.net/pipermail/security-dev/2017-May/015902.html [3] https://gist.github.com/djelinski/b4543a3eb7ea66306044c08b41bba00f -------------- next part -------------- An HTML attachment was scrubbed... URL: From abakhtin at openjdk.java.net Tue Oct 27 17:46:35 2020 From: abakhtin at openjdk.java.net (Alexey Bakhtin) Date: Tue, 27 Oct 2020 17:46:35 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore [v2] In-Reply-To: References: Message-ID: <3WEBdCUld3UbQohh2lcvjXcOOFENjH0QXqJoQycQOIE=.dfd118fa-ad26-41d2-af49-cf77ccb9e7ee@github.com> > Hi All, > > DES and DESede keys are supported by JKS/JCEKS but not supported by PKCS#12 keystores. > This issue prevents the migration of legacy applications to PKCS#12 keystore. For example, an application has some old 3DES keys that are required for certain legacy features. Java PKCS12 keystore does not support DES/3DES keys, thus, application can?t migrate to PKCS#12 > This patch adds OIDs for the DES/DESede algorithms. It is the only changes required to support DES/3DES keys in the PKCS#12 keystore. > sun/security/pkcs12/P12SecretKey test is updated to verify new secret keys in the PKCS#12 keystore. Alexey Bakhtin has updated the pull request incrementally with one additional commit since the last revision: DES oid is 1.3.14.3.2.7 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/877/files - new: https://git.openjdk.java.net/jdk/pull/877/files/09354ca6..94423b3d Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=877&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=877&range=00-01 Stats: 2 lines in 1 file changed: 0 ins; 1 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/877.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/877/head:pull/877 PR: https://git.openjdk.java.net/jdk/pull/877 From weijun at openjdk.java.net Tue Oct 27 17:50:21 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 27 Oct 2020 17:50:21 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore [v2] In-Reply-To: <3WEBdCUld3UbQohh2lcvjXcOOFENjH0QXqJoQycQOIE=.dfd118fa-ad26-41d2-af49-cf77ccb9e7ee@github.com> References: <3WEBdCUld3UbQohh2lcvjXcOOFENjH0QXqJoQycQOIE=.dfd118fa-ad26-41d2-af49-cf77ccb9e7ee@github.com> Message-ID: <7iGE7I2fFp9jB8PZ93_z_jL_iwtpDaABIbq9DaMYh-M=.5a5adade-2c3a-4fe9-a7f3-ffc54d74e647@github.com> On Tue, 27 Oct 2020 17:46:35 GMT, Alexey Bakhtin wrote: >> Hi All, >> >> DES and DESede keys are supported by JKS/JCEKS but not supported by PKCS#12 keystores. >> This issue prevents the migration of legacy applications to PKCS#12 keystore. For example, an application has some old 3DES keys that are required for certain legacy features. Java PKCS12 keystore does not support DES/3DES keys, thus, application can?t migrate to PKCS#12 >> This patch adds OIDs for the DES/DESede algorithms. It is the only changes required to support DES/3DES keys in the PKCS#12 keystore. >> sun/security/pkcs12/P12SecretKey test is updated to verify new secret keys in the PKCS#12 keystore. > > Alexey Bakhtin has updated the pull request incrementally with one additional commit since the last revision: > > DES oid is 1.3.14.3.2.7 src/java.base/share/classes/sun/security/util/KnownOIDs.java line 356: > 354: OIW_DES_CBC("1.3.14.3.2.7", "DES/CBC", "DES"), > 355: > 356: DESede("1.3.14.3.2.17", "DESede"), Please move this below before SHA-1. The items are ordered by the OIDs (within each section group). ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From abakhtin at openjdk.java.net Tue Oct 27 17:58:35 2020 From: abakhtin at openjdk.java.net (Alexey Bakhtin) Date: Tue, 27 Oct 2020 17:58:35 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore [v3] In-Reply-To: References: Message-ID: > Hi All, > > DES and DESede keys are supported by JKS/JCEKS but not supported by PKCS#12 keystores. > This issue prevents the migration of legacy applications to PKCS#12 keystore. For example, an application has some old 3DES keys that are required for certain legacy features. Java PKCS12 keystore does not support DES/3DES keys, thus, application can?t migrate to PKCS#12 > This patch adds OIDs for the DES/DESede algorithms. It is the only changes required to support DES/3DES keys in the PKCS#12 keystore. > sun/security/pkcs12/P12SecretKey test is updated to verify new secret keys in the PKCS#12 keystore. Alexey Bakhtin has updated the pull request incrementally with one additional commit since the last revision: Fix order of OIDs ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/877/files - new: https://git.openjdk.java.net/jdk/pull/877/files/94423b3d..efbd5a4f Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=877&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=877&range=01-02 Stats: 4 lines in 1 file changed: 2 ins; 2 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/877.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/877/head:pull/877 PR: https://git.openjdk.java.net/jdk/pull/877 From weijun at openjdk.java.net Tue Oct 27 18:02:19 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Tue, 27 Oct 2020 18:02:19 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore [v3] In-Reply-To: References: Message-ID: On Tue, 27 Oct 2020 17:58:35 GMT, Alexey Bakhtin wrote: >> Hi All, >> >> DES and DESede keys are supported by JKS/JCEKS but not supported by PKCS#12 keystores. >> This issue prevents the migration of legacy applications to PKCS#12 keystore. For example, an application has some old 3DES keys that are required for certain legacy features. Java PKCS12 keystore does not support DES/3DES keys, thus, application can?t migrate to PKCS#12 >> This patch adds OIDs for the DES/DESede algorithms. It is the only changes required to support DES/3DES keys in the PKCS#12 keystore. >> sun/security/pkcs12/P12SecretKey test is updated to verify new secret keys in the PKCS#12 keystore. > > Alexey Bakhtin has updated the pull request incrementally with one additional commit since the last revision: > > Fix order of OIDs Marked as reviewed by weijun (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From hchao at openjdk.java.net Tue Oct 27 18:24:22 2020 From: hchao at openjdk.java.net (Hai-May Chao) Date: Tue, 27 Oct 2020 18:24:22 GMT Subject: RFR: 8007632: DES/3DES keys support in PKCS12 keystore [v3] In-Reply-To: References: Message-ID: On Tue, 27 Oct 2020 17:59:38 GMT, Weijun Wang wrote: >> Alexey Bakhtin has updated the pull request incrementally with one additional commit since the last revision: >> >> Fix order of OIDs > > Marked as reviewed by weijun (Reviewer). Change looks good. ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From abakhtin at openjdk.java.net Tue Oct 27 18:44:20 2020 From: abakhtin at openjdk.java.net (Alexey Bakhtin) Date: Tue, 27 Oct 2020 18:44:20 GMT Subject: Integrated: 8007632: DES/3DES keys support in PKCS12 keystore In-Reply-To: References: Message-ID: On Tue, 27 Oct 2020 12:27:52 GMT, Alexey Bakhtin wrote: > Hi All, > > DES and DESede keys are supported by JKS/JCEKS but not supported by PKCS#12 keystores. > This issue prevents the migration of legacy applications to PKCS#12 keystore. For example, an application has some old 3DES keys that are required for certain legacy features. Java PKCS12 keystore does not support DES/3DES keys, thus, application can?t migrate to PKCS#12 > This patch adds OIDs for the DES/DESede algorithms. It is the only changes required to support DES/3DES keys in the PKCS#12 keystore. > sun/security/pkcs12/P12SecretKey test is updated to verify new secret keys in the PKCS#12 keystore. This pull request has now been integrated. Changeset: 7a7ce021 Author: Alexey Bakhtin Committer: Weijun Wang URL: https://git.openjdk.java.net/jdk/commit/7a7ce021 Stats: 16 lines in 2 files changed: 5 ins; 1 del; 10 mod 8007632: DES/3DES keys support in PKCS12 keystore Reviewed-by: weijun ------------- PR: https://git.openjdk.java.net/jdk/pull/877 From mcimadamore at openjdk.java.net Tue Oct 27 19:13:36 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Tue, 27 Oct 2020 19:13:36 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v17] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. > > This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). > > A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. > > A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. > > Thanks > Maurizio > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. > > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. > > After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. > > Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). > > The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. > > As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. > > In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. > > To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). > > Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). > > `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. > > The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. > > #### Memory access var handles overhaul > > The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. > > This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. > > This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. > > #### Test changes > > Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. > > [1] - https://openjdk.java.net/jeps/393 > [2] - https://openjdk.java.net/jeps/389 > [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html > [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: More 32-bit fixes for TestLayouts ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/e43f5d76..b01af093 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=16 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=15-16 Stats: 10 lines in 1 file changed: 2 ins; 4 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From github.com+10835776+stsypanov at openjdk.java.net Wed Oct 28 08:44:40 2020 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Wed, 28 Oct 2020 08:44:40 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects [v2] In-Reply-To: References: Message-ID: > As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: > @State(Scope.Thread) > @OutputTimeUnit(TimeUnit.NANOSECONDS) > @BenchmarkMode(value = Mode.AverageTime) > public class AtomicBenchmark { > @Benchmark > public Object defaultValue() { > return new AtomicInteger(); > } > @Benchmark > public Object explicitValue() { > return new AtomicInteger(0); > } > } > THis benchmark demonstrates that `explicitValue()` is much slower: > Benchmark Mode Cnt Score Error Units > AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op > AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op > So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. > > I've tested the changes locally, both tier1 and tier 2 are ok. > > Could one create an issue for tracking this? ?????? ??????? has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains one additional commit since the last revision: 8255299: Drop explicit zeroing at instantiation of Atomic* objects ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/818/files - new: https://git.openjdk.java.net/jdk/pull/818/files/c1fb362f..7dc646d0 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=818&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=818&range=00-01 Stats: 4576 lines in 201 files changed: 2659 ins; 1135 del; 782 mod Patch: https://git.openjdk.java.net/jdk/pull/818.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/818/head:pull/818 PR: https://git.openjdk.java.net/jdk/pull/818 From github.com+10835776+stsypanov at openjdk.java.net Wed Oct 28 08:44:41 2020 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Wed, 28 Oct 2020 08:44:41 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects [v2] In-Reply-To: References: Message-ID: On Sat, 24 Oct 2020 23:12:09 GMT, Phil Race wrote: >> ?????? ??????? has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains one additional commit since the last revision: >> >> 8255299: Drop explicit zeroing at instantiation of Atomic* objects > > client changes are fine Rebased onto master to have the fix introduced in https://github.com/openjdk/jdk/pull/778 ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From serb at openjdk.java.net Wed Oct 28 08:52:19 2020 From: serb at openjdk.java.net (Sergey Bylokhov) Date: Wed, 28 Oct 2020 08:52:19 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects [v2] In-Reply-To: References: Message-ID: On Wed, 28 Oct 2020 08:40:02 GMT, ?????? ??????? wrote: >> client changes are fine > > Rebased onto master to have the fix introduced in https://github.com/openjdk/jdk/pull/778 FYI it is better to use merge, instead of rebase+force push. Rebase breaks history and all existed code comments. ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From github.com+10835776+stsypanov at openjdk.java.net Wed Oct 28 08:59:23 2020 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Wed, 28 Oct 2020 08:59:23 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects [v2] In-Reply-To: References: Message-ID: On Wed, 28 Oct 2020 08:49:38 GMT, Sergey Bylokhov wrote: >> Rebased onto master to have the fix introduced in https://github.com/openjdk/jdk/pull/778 > > FYI it is better to use merge, instead of rebase+force push. Rebase breaks history and all existed code comments. @mrserb thanks for pointing this out! ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From ihse at openjdk.java.net Wed Oct 28 10:55:27 2020 From: ihse at openjdk.java.net (Magnus Ihse Bursie) Date: Wed, 28 Oct 2020 10:55:27 GMT Subject: RFR: 8235710: Remove the legacy elliptic curves [v2] In-Reply-To: <8nr0RrYr-7cnjo4sSs9BpMWPCfFptjBDVu56hoDxl0U=.d5addb7e-5646-4f58-a5e7-29eda6cae1fc@github.com> References: <8nr0RrYr-7cnjo4sSs9BpMWPCfFptjBDVu56hoDxl0U=.d5addb7e-5646-4f58-a5e7-29eda6cae1fc@github.com> Message-ID: On Tue, 22 Sep 2020 14:04:55 GMT, Sean Mullan wrote: >> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision: >> >> remove JDKOPT_DETECT_INTREE_EC from configure.ac > > throw new IllegalStateException( > new InvalidAlgorithmParameterException( > "Curve not supported: Private: " + > ((privNC != null) ? privNC.toString() : " unknown") + > ", PublicKey:" + > ((pubNC != null) ? pubNC.toString() : " unknown"))); I opened https://bugs.openjdk.java.net/browse/JDK-8255530 for the remaining cleanup. ------------- PR: https://git.openjdk.java.net/jdk/pull/289 From github.com+10835776+stsypanov at openjdk.java.net Wed Oct 28 12:14:48 2020 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Wed, 28 Oct 2020 12:14:48 GMT Subject: Integrated: 8255299: Drop explicit zeroing at instantiation of Atomic* objects In-Reply-To: References: Message-ID: On Thu, 22 Oct 2020 20:46:15 GMT, ?????? ??????? wrote: > As discussed in https://github.com/openjdk/jdk/pull/510 there is never a reason to explicitly instantiate any instance of `Atomic*` class with its default value, i.e. `new AtomicInteger(0)` could be replaced with `new AtomicInteger()` which is faster: > @State(Scope.Thread) > @OutputTimeUnit(TimeUnit.NANOSECONDS) > @BenchmarkMode(value = Mode.AverageTime) > public class AtomicBenchmark { > @Benchmark > public Object defaultValue() { > return new AtomicInteger(); > } > @Benchmark > public Object explicitValue() { > return new AtomicInteger(0); > } > } > THis benchmark demonstrates that `explicitValue()` is much slower: > Benchmark Mode Cnt Score Error Units > AtomicBenchmark.defaultValue avgt 30 4.778 ? 0.403 ns/op > AtomicBenchmark.explicitValue avgt 30 11.846 ? 0.273 ns/op > So meanwhile https://bugs.openjdk.java.net/browse/JDK-8145948 is still in progress we could trivially replace explicit zeroing with default constructors gaining some performance benefit with no risk. > > I've tested the changes locally, both tier1 and tier 2 are ok. > > Could one create an issue for tracking this? This pull request has now been integrated. Changeset: 3c4fc793 Author: Sergey Tsypanov Committer: Daniel Fuchs URL: https://git.openjdk.java.net/jdk/commit/3c4fc793 Stats: 19 lines in 17 files changed: 0 ins; 3 del; 16 mod 8255299: Drop explicit zeroing at instantiation of Atomic* objects Reviewed-by: redestad, serb, prr ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From dfuchs at openjdk.java.net Wed Oct 28 12:14:47 2020 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Wed, 28 Oct 2020 12:14:47 GMT Subject: RFR: 8255299: Drop explicit zeroing at instantiation of Atomic* objects [v2] In-Reply-To: References: Message-ID: On Wed, 28 Oct 2020 08:56:05 GMT, ?????? ??????? wrote: >> FYI it is better to use merge, instead of rebase+force push. Rebase breaks history and all existed code comments. > > @mrserb thanks for pointing this out! Thanks for updating with latest master changes Sergey! My tests were all green. ------------- PR: https://git.openjdk.java.net/jdk/pull/818 From ascarpino at openjdk.java.net Wed Oct 28 17:37:52 2020 From: ascarpino at openjdk.java.net (Anthony Scarpino) Date: Wed, 28 Oct 2020 17:37:52 GMT Subject: RFR: 8235710: Remove the legacy elliptic curves [v2] In-Reply-To: References: <8nr0RrYr-7cnjo4sSs9BpMWPCfFptjBDVu56hoDxl0U=.d5addb7e-5646-4f58-a5e7-29eda6cae1fc@github.com> Message-ID: On Wed, 28 Oct 2020 10:52:27 GMT, Magnus Ihse Bursie wrote: >> throw new IllegalStateException( >> new InvalidAlgorithmParameterException( >> "Curve not supported: Private: " + >> ((privNC != null) ? privNC.toString() : " unknown") + >> ", PublicKey:" + >> ((pubNC != null) ? pubNC.toString() : " unknown"))); > > I opened https://bugs.openjdk.java.net/browse/JDK-8255530 for the remaining cleanup. I have the change in a workspace, just hadn't created the bug yet.. thanks ------------- PR: https://git.openjdk.java.net/jdk/pull/289 From weijun at openjdk.java.net Wed Oct 28 21:01:52 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Wed, 28 Oct 2020 21:01:52 GMT Subject: RFR: 8255536: Remove the directsign property and option Message-ID: I regret adding the `directsign` property/option to JarSigner/jarsigner. See the bug for details. ------------- Commit messages: - 8255536: Remove the directsign property and option Changes: https://git.openjdk.java.net/jdk/pull/915/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=915&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8255536 Stats: 184 lines in 7 files changed: 14 ins; 162 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/915.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/915/head:pull/915 PR: https://git.openjdk.java.net/jdk/pull/915 From weijun at openjdk.java.net Wed Oct 28 21:07:55 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Wed, 28 Oct 2020 21:07:55 GMT Subject: RFR: 8255494: PKCS7 should use digest algorithm to verify the signature Message-ID: This is a regression made by [JDK-8242068](https://bugs.openjdk.java.net/browse/JDK-8242068). When the digest algorithm is not the same as the hash part of the signature algorithm, we used to combine the digest algorithm with the key part of the signature algorithm into a new signature algorithm and use it when generating a signature. The previous code change uses the signature algorithm in the SignerInfo directly. This bugfix will revert to the old behavior. ------------- Commit messages: - 8255494: PKCS7 should use digest algorithm to verify the signature Changes: https://git.openjdk.java.net/jdk/pull/916/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=916&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8255494 Stats: 126 lines in 3 files changed: 113 ins; 5 del; 8 mod Patch: https://git.openjdk.java.net/jdk/pull/916.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/916/head:pull/916 PR: https://git.openjdk.java.net/jdk/pull/916 From valeriep at openjdk.java.net Wed Oct 28 21:40:49 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Wed, 28 Oct 2020 21:40:49 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files Message-ID: Could someone please help review this PKCS#11 v3.0 header files update? Changes are straight-forward as below: 1) Updated pkcs11.h, pkcs11f.h, pkcs11t.h to v3.0 2) Updated java side w/ the new constants definitions and name/error code mappings. For the native headers, it's a direct copy of the official v3.0 headers except that I have to remove the tab space, and trailing white spaces due to JDK code requirement. I verified the result using 'diff -w'. As for the java side, the edit is based on the diff of native headers. I also commented out some of the unused native identifiers at java side. I am adding the SHA-3 digests, signatures, and macs in a separate RFE and would need this one to be reviewed/integrated first. Thanks, Valerie ------------- Commit messages: - 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files Changes: https://git.openjdk.java.net/jdk/pull/917/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=917&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8244154 Stats: 1592 lines in 6 files changed: 1146 ins; 52 del; 394 mod Patch: https://git.openjdk.java.net/jdk/pull/917.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/917/head:pull/917 PR: https://git.openjdk.java.net/jdk/pull/917 From hchao at openjdk.java.net Thu Oct 29 02:20:44 2020 From: hchao at openjdk.java.net (Hai-May Chao) Date: Thu, 29 Oct 2020 02:20:44 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Wed, 28 Oct 2020 21:35:25 GMT, Valerie Peng wrote: > Could someone please help review this PKCS#11 v3.0 header files update? > > Changes are straight-forward as below: > 1) Updated pkcs11.h, pkcs11f.h, pkcs11t.h to v3.0 > 2) Updated java side w/ the new constants definitions and name/error code mappings. > > For the native headers, it's a direct copy of the official v3.0 headers except that I have to remove the tab space, and trailing white spaces due to JDK code requirement. I verified the result using 'diff -w'. As for the java side, the edit is based on the diff of native headers. I also commented out some of the unused native identifiers at java side. > > I am adding the SHA-3 digests, signatures, and macs in a separate RFE and would need this one to be reviewed/integrated first. > > Thanks, > Valerie Changes look good. Only minor comments. src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/wrapper/Functions.java line 793: > 791: addMech(CKM_SHA3_512_RSA_PKCS_PSS, "CKM_SHA3_512_RSA_PKCS_PSS"); > 792: addMech(CKM_SHA3_224_RSA_PKCS, "CKM_SHA3_224_RSA_PKCS"); > 793: addMech(CKM_SHA3_224_RSA_PKCS_PSS, "CKM_SHA3_224_RSA_PKCS_PSS"); It appears that you're arranging the addMech(with CKM_xxx) based on the mechanism values. How about the code from #773 to #793, move it up? src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/wrapper/Functions.java line 1095: > 1093: addMech(CKM_SP800_108_FEEDBACK_KDF, "CKM_SP800_108_FEEDBACK_KDF"); > 1094: addMech(CKM_SP800_108_DOUBLE_PIPELINE_KDF, > 1095: "CKM_SP800_108_DOUBLE_PIPELINE_KDF"); same comment as above. src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/wrapper/PKCS11Constants.java line 987: > 985: public static final long CKM_SP800_108_FEEDBACK_KDF = 0x000003adL; > 986: public static final long CKM_SP800_108_DOUBLE_PIPELINE_KDF = 0x000003aeL; > 987: Same comment. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From weijun at openjdk.java.net Thu Oct 29 03:21:42 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 29 Oct 2020 03:21:42 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 02:18:00 GMT, Hai-May Chao wrote: >> Could someone please help review this PKCS#11 v3.0 header files update? >> >> Changes are straight-forward as below: >> 1) Updated pkcs11.h, pkcs11f.h, pkcs11t.h to v3.0 >> 2) Updated java side w/ the new constants definitions and name/error code mappings. >> >> For the native headers, it's a direct copy of the official v3.0 headers except that I have to remove the tab space, and trailing white spaces due to JDK code requirement. I verified the result using 'diff -w'. As for the java side, the edit is based on the diff of native headers. I also commented out some of the unused native identifiers at java side. >> >> I am adding the SHA-3 digests, signatures, and macs in a separate RFE and would need this one to be reviewed/integrated first. >> >> Thanks, >> Valerie > > Changes look good. Only minor comments. Just curious, can the Java files be generated during the build process? ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From XUELEI.FAN at ORACLE.COM Thu Oct 29 03:40:34 2020 From: XUELEI.FAN at ORACLE.COM (Xue-Lei Fan) Date: Wed, 28 Oct 2020 20:40:34 -0700 Subject: Request for comment: the session ticket protection scheme for distributed TLS sessions. In-Reply-To: References: Message-ID: The PNG may be too large to open from some mail system. Here is the PDF version. BTW, I also made an update on the use of AEAD algorithm with additional data. Thanks, Xuelei -------------- next part -------------- A non-text attachment was scrubbed... Name: distributed_credential_protection.pdf Type: application/pdf Size: 95691 bytes Desc: not available URL: -------------- next part -------------- > On Oct 23, 2020, at 8:58 AM, Xuelei Fan wrote: > > Hi, > > I'm working on the JEP to improve the scalability and throughput of the TLS implementation, by supporting distributed session resumption across clusters of computers. > > TLS session tickets will used for session resumption in a cluster. To support distributed session resumption, a session ticket that is generated and protected in one server node must be usable for session resumption on other server nodes in the distributed system. Each node should use the same session ticket structure, and share the secrets that are used to protect session tickets. More details, please refer to the JEP: > https://bugs.openjdk.java.net/browse/JDK-8245551 > > It is a essential part of the implementation that we need to define a session ticket protection scheme. The scheme will support key generation, key rotation and key synchronization across clusters of computers. > > The attached doc_distributed_credential_protection.md is a markdown file, which may not easy to read. So I attached a rendered picture as well. > > Please let me know if you have any concerns. Any comments are welcome. > > Thanks, > Xuelei > From mcimadamore at openjdk.java.net Thu Oct 29 14:13:03 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 29 Oct 2020 14:13:03 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v18] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. > > This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). > > A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. > > A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. > > Thanks > Maurizio > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. > > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. > > After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. > > Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). > > The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. > > As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. > > In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. > > To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). > > Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). > > `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. > > The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. > > #### Memory access var handles overhaul > > The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. > > This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. > > This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. > > #### Test changes > > Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. > > [1] - https://openjdk.java.net/jeps/393 > [2] - https://openjdk.java.net/jeps/389 > [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html > [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Fix issues with derived buffers and IO operations ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/548/files - new: https://git.openjdk.java.net/jdk/pull/548/files/b01af093..e3ec6b4c Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=17 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=16-17 Stats: 81 lines in 5 files changed: 77 ins; 0 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Thu Oct 29 14:16:49 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Thu, 29 Oct 2020 14:16:49 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) In-Reply-To: References: Message-ID: On Tue, 27 Oct 2020 14:40:29 GMT, Maurizio Cimadamore wrote: >> This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: >> >> * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads >> * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually >> * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. >> >> A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. >> >> This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). >> >> A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. >> >> A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. >> >> Thanks >> Maurizio >> >> Javadoc: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html >> >> Specdiff: >> >> http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html >> >> CSR: >> >> https://bugs.openjdk.java.net/browse/JDK-8254163 >> >> >> >> ### API Changes >> >> * `MemorySegment` >> * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) >> * added a no-arg factory for a native restricted segment representing entire native heap >> * rename `withOwnerThread` to `handoff` >> * add new `share` method, to create shared segments >> * add new `registerCleaner` method, to register a segment against a cleaner >> * add more helpers to create arrays from a segment e.g. `toIntArray` >> * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) >> * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) >> * `MemoryAddress` >> * drop `segment` accessor >> * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment >> * `MemoryAccess` >> * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). >> * `MemoryHandles` >> * drop `withOffset` combinator >> * drop `withStride` combinator >> * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. >> * `Addressable` >> * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. >> * `MemoryLayouts` >> * A new layout, for machine addresses, has been added to the mix. >> >> >> >> ### Implementation changes >> >> There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. >> >> #### Shared segments >> >> The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. >> >> After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. >> >> Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). >> >> The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. >> >> As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. >> >> In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. >> >> To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). >> >> Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). >> >> `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. >> >> The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. >> >> #### Memory access var handles overhaul >> >> The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. >> >> This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. >> >> This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. >> >> #### Test changes >> >> Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. >> >> [1] - https://openjdk.java.net/jeps/393 >> [2] - https://openjdk.java.net/jeps/389 >> [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html >> [4] - https://openjdk.java.net/jeps/312 > >> @mcimadamore, if you pull from current master, you would get the Linux x86_32 tier1 run "for free". > > Just did that - I also removed TestMismatch from the problem list in the latest iteration, and fixed the alignment for long/double layouts, after chatting with the team (https://bugs.openjdk.java.net/browse/JDK-8255350) I've just uploaded another iteration which addresses some comments from @AlanBateman. Basically, there are some operations on Channel and Socket which take ByteBuffer as arguments, and then, if such buffers are *direct*, they get the address and pass it down to some native function. This idiom is problematic because there's no way to guarantee that the buffer won't be closed (if obtained from a memory segment) after the address has been obtained. As a stop gap solution, I've introduced checks in `DirectBuffer::address` method, which is used in around 30 places in the JDK. This method will now throw if (a) the buffer has a shared scope, or (b) if the scope is confined, but already closed. With this extra check, I believe there's no way to misuse the buffer obtained from a segment. We have discussed plans to remove this limitations (which we think will be possible) - but for the time being, it's better to play the conservative card. ------------- PR: https://git.openjdk.java.net/jdk/pull/548 From xuelei.fan at oracle.com Fri Oct 23 15:58:49 2020 From: xuelei.fan at oracle.com (Xuelei Fan) Date: Fri, 23 Oct 2020 08:58:49 -0700 Subject: Request for comment: the session ticket protection scheme for distributed TLS sessions. Message-ID: Hi, I'm working on the JEP to improve the scalability and throughput of the TLS implementation, by supporting distributed session resumption across clusters of computers. TLS session tickets will used for session resumption in a cluster. To support distributed session resumption, a session ticket that is generated and protected in one server node must be usable for session resumption on other server nodes in the distributed system. Each node should use the same session ticket structure, and share the secrets that are used to protect session tickets. More details, please refer to the JEP: https://bugs.openjdk.java.net/browse/JDK-8245551 It is a essential part of the implementation that we need to define a session ticket protection scheme. The scheme will support key generation, key rotation and key synchronization across clusters of computers. The attached doc_distributed_credential_protection.md is a markdown file, which may not easy to read. So I attached a rendered picture as well. Please let me know if you have any concerns. Any comments are welcome. Thanks, Xuelei -------------- next part -------------- A non-text attachment was scrubbed... Name: distributed-credentials.png Type: image/png Size: 471611 bytes Desc: not available URL: -------------- next part -------------- # Distributed Credential Protection ## Summary Design a distributed credential protecion scheme so that the credential generated and proected in one server node could be used in an entire cluster. ## Motivation For TLS protocols, an initial connection will negotiate the security parameters and then establishe the security channel between client and server. The initial connection is expensive because a lot of cryptographic operations are involved. The negotiated parameters could be cached in a protected credential. The credential could be reused for subsequent connections. The credential reuse could significantly improve the performance the the subsequent connections. We want to extend the benefit from connections between the same client and server to connections between the same client and an entire cluster. Although the credential is created and protected in one server node, it must be usable on any server node in the cluster. As require that each node should share the secrets/keys that are used to protec the credential. The distributed credential protection scheme should take care of the key genetation, key rotation and synchronization across the clusters of computers. ## Related Cryptographic Algorithms and Terms ### Secure hash function A hash function is any function that can be used to map data of arbitrary size to fixed-size values. A secure hash algorithm is a hash function that is suitable for use in cryptography. ``` HASH(m) -> MD Options: Hash a hash function; HashLen donotes the length of the hash function output in octets. Input: m the message Output: MD a fixed-size message digest (of HashLen octets) The output MD is calculated by mapping the input message to fixed-size values with specific hash algorithm. The output could be notated as follows: MD = HASH(m) ``` ### Keyed-Hash Message Authentication Code (HMAC) In cryptography, an HMAC(Keyed-hash message authentication code) is a specific type of message authentication code (MAC) involving a cryptographic hash function and a secret cryptographic key. ``` HMAC-Hash(m, K) -> MAC Options: Hash a hash function; HashLen donotes the length of the hash function output in octets. Input: m the message K the secret key Output: MAC a fixed-size message authentication code (of HashLen octets) The output MAC could be notated as follows: MAC = HMAC-Hash(m, K) ``` ### HMAC-based key derivation function (HKDF) HKDF is a HMAC-based key derivation function (KDF). It is used to take some source of initial keying material and derive from it one or more cryptographically strong secret keys. HKDF follows the "extract-then-expand" paradigm, where the KDF logically consists of two modules. The first stage takes the input keying material and "extracts" from it a fixed-length pseudorandom key K. The second stage "expands" the key K into several additional pseudorandom keys (the output of the KDF). 1. HKDF-Extract(salt, IKM) -> PRK ``` Options: Hash a hash function; HashLen denotes the length of the hash function output in octets Inputs: salt optional salt value (a non-secret random value); if not provided, it is set to a string of HashLen zeros. IKM input keying material Output: PRK a pseudorandom key (of HashLen octets) The output PRK is calculated as follows: PRK = HMAC-Hash(salt, IKM) or notated as follows: PRK = HKDF-Extract(salt, IKM) ``` 2. HKDF-Expand(PRK, info, L) -> OKM ``` Options: Hash a hash function; HashLen denotes the length of the hash function output in octets Inputs: PRK a pseudorandom key of at least HashLen octets (usually, the output from the extract step) info optional context and application specific information (can be a zero-length string) L length of output keying material in octets (<= 255*HashLen) Output: OKM output keying material (of L octets) The output OKM could be notated as follows: OKM = HKDF-Expand(PRK, info, L) ``` ### Authenticated encryption with associated data (AEAD) Authenticated encryption with associated data (AEAD) is a form of encryption which simultaneously assure the confidentiality and authenticity of data. ``` AEAD-Encrypt(key, nonce, additional_data, plaintext) -> AEADEncrypted Input: key the secret key for the encryption nonce a unique value for each encryption operation additional_data the associated data plaintext the message to be encrypted Output: AEADEncrypted the ciphertext AEAD-Decrypt(key, nonce, additional_data, AEADEncrypted) -> plaintext Input: key the secret key for the encryption nonce a unique value for each encryption operation additional_data the associated data AEADEncrypted the ciphertext Output: plaintext the decrypted message ``` ## Distributed Credential Protecion Scheme The basic idea of this proposal is deriving credential protecion key from the server authentication possessions. In general, the server possesses some private information for the server authentication on TLS connections, for example, the RSA/EC private key. Every server in the distributed system should be able to access the possession. With the proposed scheme, the credential protecion keys will be automatically generated and rotated in each server, and automatically synchronized among the distributed system. This proposal can be considered as a triple stage process. 1. Select the server possessions for key generation; 2. Design the key rotation scheme; 3. Design the credential protecion scheme. ### Select the possessions and derive the master key derivation key. Each node in the cluster should possesses some private information for server authentication. In TLS context, we select to used the private key and public key as the server possessions, which is the same in each server node in the cluster. ``` Possession: encoded public key | encoded private key ``` The server possessions cannot be used directly for credential protection. Instead, the keying material should be generated from the server possessions. ``` Option: Hash: hash algorithm Label: a choosen octests label for key derivation Input: Possession: the server possession // Derive the shared key materials from the server possession SKM = HASH(Possession || SKMLabel) Hash: the hash algorithm SKMLabel: label for this derivation // Derive the master key derivation key. KDK = HKDF_Extract(KDKLabel, SKM) KDKLabel: label for this derivation, as salf for HKDF_Extract // Clean the shared key materials SKM = zeros ``` ```mermaid graph TD A[server possession] --> C{hash algorithm} B[possession label] --> C{hash algorithm} C --> D[shared key materials] D --> F{key derivation algorithm} E[key derivation label] --> F{key derivation algorithm} F --> G[the master key derivation key] ``` ### Time based key rotation scheme If the maximum key use is limited, or could be exceeded, the key should be rotated before reach the limit. At a time, each node in the cluster should use the credential protection key. The secret key should be updated and synchronized among the cluster. A key rotation scheme defines the key update and synchronization policy so that the credential protection keys do not exceed the maximum use-limit or the timeout limit. The time based key rotation schme based on the follow two assumptions: 1. The system clock in each node should be synchronized. It is doable by using the Network Time Protocol (NTP). 2. If the system clock is not precise synchronized, the secure parameters could be re-negotiated, without breaking the connection. Here is the time-based key rotation schme: 1. Define the key rotation timeout, for example one week, or two weeks. ``` private static final long KEY_DERIVATION_PERIOD = TimeUnit.DAYS.toMillis(7); ``` 2. When a node inserted into the system, calculate how many periods (or timeouts) have passed since a past-time (for example, 01/01/1970). ``` long periodsSince1970 = System.currentTimeMillis() / KEY_DERIVATION_PERIOD; ``` 3. Derive the secret key with a deterministic key derivation algorithm (for example, HKDF). ``` Option: Hash: hash algorithm used for the key derivation Input: PRK a pseudorandom key, which is derived from the server possessions. info the periods since a past time. L length of output keying material in octets. // Derive the credential encryption key. TEK = HKDF-Expand(PRK, info, L) ``` ``` SecretKey credentialEncryptionKey = HKDF.of(scheme.hashAlg).expand( ppk, Utilities.toByteArray(periodsSince1970), keyScheme.keySize, keyScheme.keyAlg); ``` 4. When the system clock moving onto the next period, update the credential encryption key, as described in #3. If combining the server possession part together, the scenarios is as showned in the following diagram: ```mermaid graph TD A[server possession] --> C{hash algorithm} B[possession label] --> C{hash algorithm} C --> D[shared key materials] D --> F{key derivation algorithm} E[key derivation label] --> F{key derivation algorithm} F --> G[the master key derivation key] G --> I{key derivation algorithm} H[the periods since a past time] -. rotate .-> H H[the periods since a past time] --> I{key derivation algorithm} I --> J[the credential encryption key] ``` ### Credential protecion scheme The protected credential is defined as: ``` Credential: periods | nonce | EncryptedParameters periods : how many periods (or timeouts) have passed since a past-time nonce : a random number for each encryption (for AEAD cipher) EncryptedParameters : encrypted negotiated-parameters for the initial connection ``` Here is the scheme for the credential creation: ``` Input: key : the credential encryption key nonce : a random number for each encryption additional_data : empty or a hard-coded label. plaintextParameters : plaintext parameters before encryption. periods : the peirods have passed since a past-time. Output: the protected credential // Protect the negotiated parameters EncryptedParameters = AEAD-Encrypt(key, nonce, additional_data, plaintextParameters) // Construct the protected credential ProtectedCredential = periods | nonce | EncryptedParameters ``` Here is the scheme for credential reuse: ``` Input: additional_data : empty or a hard-coded label. ProtectedCredential : the protected credential Output: plaintext parameters before encryption, and the peirods have passed since a past-time // Decapsulate the protected credential, and get the peirods, nonce and EncryptedParameters. periods | nonce | EncryptedParameters // Get the key specified by the periods. ondutyKey = ... // Rotate the key if needed. // Decrypt the EncryptedParameters plaintextParameters = AEAD-Decrypt(ondutyKey, nonce, additional_data, EncryptedParameters) ``` From clanger at openjdk.java.net Thu Oct 29 15:20:55 2020 From: clanger at openjdk.java.net (Christoph Langer) Date: Thu, 29 Oct 2020 15:20:55 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 Message-ID: It seems that there exists a memory/performance regression that was introduced with JDK-8210985: Update the default SSL session cache size to 20480. The idea to limit the maixmum SSL session cache size by itself is good. Unfortunately, as per the current implementation of sun.security.util.MemoryCache, it also modifies the initial size of the LinkedHashMap that is backing the cache to initialize with more than the maximum size. I suggest to restore the original behavior that initializes with an initialCapacity of 1 in most cases. That was true when before JDK-8210985, the property javax.net.ssl.sessionCacheSize was not set at all. In case it was set, the initial size would have been like now, (javax.net.ssl.sessionCacheSize / 0.75f) + 1, which still seems strange. ------------- Commit messages: - Copyright year - JDK-8255603: Memory/Performance regression after JDK-8210985 Changes: https://git.openjdk.java.net/jdk/pull/937/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=937&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8255603 Stats: 3 lines in 1 file changed: 0 ins; 1 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/937.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/937/head:pull/937 PR: https://git.openjdk.java.net/jdk/pull/937 From xuelei at openjdk.java.net Thu Oct 29 15:56:42 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Thu, 29 Oct 2020 15:56:42 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 15:11:09 GMT, Christoph Langer wrote: > It seems that there exists a memory/performance regression that was introduced with JDK-8210985: Update the default SSL session cache size to 20480. > > The idea to limit the maixmum SSL session cache size by itself is good. Unfortunately, as per the current implementation of sun.security.util.MemoryCache, it also modifies the initial size of the LinkedHashMap that is backing the cache to initialize with more than the maximum size. > > I suggest to restore the original behavior that initializes with an initialCapacity of 1 in most cases. That was true when before JDK-8210985, the property javax.net.ssl.sessionCacheSize was not set at all. > In case it was set, the initial size would have been like now, (javax.net.ssl.sessionCacheSize / 0.75f) + 1, which still seems strange. Did you have a benchmark with various cache sizes (for example, from 1 to 10K) and various connections (for example from 1 to 10K) for those components (including TLS implementation) that use Cache? ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From clanger at openjdk.java.net Thu Oct 29 16:03:48 2020 From: clanger at openjdk.java.net (Christoph Langer) Date: Thu, 29 Oct 2020 16:03:48 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 15:54:27 GMT, Xue-Lei Andrew Fan wrote: > Did you have a benchmark with various cache sizes (for example, from 1 to 10K) and various connections (for example from 1 to 10K) for those components (including TLS implementation) that use Cache? Nope, we've just seen the memory regression in a certain customer use case (lot's of meory retained by a finalizer) and confirmed it resolved after setting javax.net.ssl.sessionCacheSize to 0. But I guess this change merits certain further benchmarking to get it right. ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From simonis at openjdk.java.net Thu Oct 29 16:24:49 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Thu, 29 Oct 2020 16:24:49 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 16:01:19 GMT, Christoph Langer wrote: >> Did you have a benchmark with various cache sizes (for example, from 1 to 10K) and various connections (for example from 1 to 10K) for those components (including TLS implementation) that use Cache? > >> Did you have a benchmark with various cache sizes (for example, from 1 to 10K) and various connections (for example from 1 to 10K) for those components (including TLS implementation) that use Cache? > > Nope, we've just seen the memory regression in a certain customer use case (lot's of meory retained by a finalizer) and confirmed it resolved after setting javax.net.ssl.sessionCacheSize to 0. > > But I guess this change merits certain further benchmarking to get it right. Congratulations! You beat me about half an hour with this issue :) I was debugging this regression already the whole day and when I searched for JDK-8210985 in my mailbox I saw your post. ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From xuelei at openjdk.java.net Thu Oct 29 16:24:49 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Thu, 29 Oct 2020 16:24:49 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 16:01:19 GMT, Christoph Langer wrote: > > Did you have a benchmark with various cache sizes (for example, from 1 to 10K) and various connections (for example from 1 to 10K) for those components (including TLS implementation) that use Cache? > > Nope, we've just seen the memory regression in a certain customer use case (lot's of meory retained by a finalizer) and confirmed it resolved after setting javax.net.ssl.sessionCacheSize to 0. > Did you have this patch checked with the customer? I think the performance may be similar or improved comparing to set the cache size to 0. > But I guess this change merits certain further benchmarking to get it right. > It looks good to me, but we may be more confident with it if there is a benchmarking. ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From clanger at openjdk.java.net Thu Oct 29 16:47:42 2020 From: clanger at openjdk.java.net (Christoph Langer) Date: Thu, 29 Oct 2020 16:47:42 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 16:21:47 GMT, Xue-Lei Andrew Fan wrote: >>> Did you have a benchmark with various cache sizes (for example, from 1 to 10K) and various connections (for example from 1 to 10K) for those components (including TLS implementation) that use Cache? >> >> Nope, we've just seen the memory regression in a certain customer use case (lot's of meory retained by a finalizer) and confirmed it resolved after setting javax.net.ssl.sessionCacheSize to 0. >> >> But I guess this change merits certain further benchmarking to get it right. > >> > Did you have a benchmark with various cache sizes (for example, from 1 to 10K) and various connections (for example from 1 to 10K) for those components (including TLS implementation) that use Cache? >> >> Nope, we've just seen the memory regression in a certain customer use case (lot's of meory retained by a finalizer) and confirmed it resolved after setting javax.net.ssl.sessionCacheSize to 0. >> > Did you have this patch checked with the customer? I think the performance may be similar or improved comparing to set the cache size to 0. > >> But I guess this change merits certain further benchmarking to get it right. >> > It looks good to me, but we may be more confident with it if there is a benchmarking. We're currently rolling out a patch to our SAP JVM shipment (based on Oracle's JDK 8 licensee repository) with exactly this content. We will then check with the customer but I'd suspect his results will about the same as with -Djavax.net.ssl.sessionCacheSize=0. If you require some benchmarking I guess it'll take me some more time. In the end I doubt that we'll find a better default value than 1 for the cache size as it's hard to predict how full a cache will be in the average. Maybe one could spend a property for the initial size - but I also doubt that's worth the effort. I also think that for the use case in StatusResponseManager's responseCache the influence of a different initial value is neglectable. ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From simonis at openjdk.java.net Thu Oct 29 16:55:42 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Thu, 29 Oct 2020 16:55:42 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 16:45:01 GMT, Christoph Langer wrote: >>> > Did you have a benchmark with various cache sizes (for example, from 1 to 10K) and various connections (for example from 1 to 10K) for those components (including TLS implementation) that use Cache? >>> >>> Nope, we've just seen the memory regression in a certain customer use case (lot's of meory retained by a finalizer) and confirmed it resolved after setting javax.net.ssl.sessionCacheSize to 0. >>> >> Did you have this patch checked with the customer? I think the performance may be similar or improved comparing to set the cache size to 0. >> >>> But I guess this change merits certain further benchmarking to get it right. >>> >> It looks good to me, but we may be more confident with it if there is a benchmarking. > > We're currently rolling out a patch to our SAP JVM shipment (based on Oracle's JDK 8 licensee repository) with exactly this content. We will then check with the customer but I'd suspect his results will about the same as with -Djavax.net.ssl.sessionCacheSize=0. > > If you require some benchmarking I guess it'll take me some more time. > > In the end I doubt that we'll find a better default value than 1 for the cache size as it's hard to predict how full a cache will be in the average. Maybe one could spend a property for the initial size - but I also doubt that's worth the effort. > > I also think that for the use case in StatusResponseManager's responseCache the influence of a different initial value is neglectable. I can confirm that JDK-8210985 can cause massive regressions in memory consumption. Also, [JDK-8253116 Performance regression observed post upgrade to 8u261](https://bugs.openjdk.java.net/browse/JDK-8253116) is clearly a duplicate for this. I'm fine with setting the initial cache size to 1 as this restores the original behavior for `javax.net.ssl.sessionCacheSize=0`. The other possibility would be to use the default size for `LinkedHashMap` but that's not easy if we want to set the `accessOrder` because the only constructor which takes the `accessOrder` also requires the specification of `initialCapacitiy` and `loadFactor`. Otherwise, `LinkedHashMap`s capacity defaults to `HashMap`s default capacity which is `16`. Setting the `initialCapacity` to `1` will initialize the hash map with one bucket (see initialization code in `HashMap`). /** * Returns a power of two size for the given target capacity. */ static final int tableSizeFor(int cap) { int n = cap - 1; n |= n >>> 1; n |= n >>> 2; n |= n >>> 4; n |= n >>> 8; n |= n >>> 16; return (n < 0) ? 1 : (n >= MAXIMUM_CAPACITY) ? MAXIMUM_CAPACITY : n + 1; } Benchmarking is probably hard because we don't know the average occupancy of the map. So in the end I think `1` is a good solution for now. ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From simonis at openjdk.java.net Thu Oct 29 17:08:44 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Thu, 29 Oct 2020 17:08:44 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 15:11:09 GMT, Christoph Langer wrote: > It seems that there exists a memory/performance regression that was introduced with JDK-8210985: Update the default SSL session cache size to 20480. > > The idea to limit the maixmum SSL session cache size by itself is good. Unfortunately, as per the current implementation of sun.security.util.MemoryCache, it also modifies the initial size of the LinkedHashMap that is backing the cache to initialize with more than the maximum size. > > I suggest to restore the original behavior that initializes with an initialCapacity of 1 in most cases. That was true when before JDK-8210985, the property javax.net.ssl.sessionCacheSize was not set at all. > In case it was set, the initial size would have been like now, (javax.net.ssl.sessionCacheSize / 0.75f) + 1, which still seems strange. Marked as reviewed by simonis (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From xuelei at openjdk.java.net Thu Oct 29 17:16:45 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Thu, 29 Oct 2020 17:16:45 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 15:11:09 GMT, Christoph Langer wrote: > It seems that there exists a memory/performance regression that was introduced with JDK-8210985: Update the default SSL session cache size to 20480. > > The idea to limit the maixmum SSL session cache size by itself is good. Unfortunately, as per the current implementation of sun.security.util.MemoryCache, it also modifies the initial size of the LinkedHashMap that is backing the cache to initialize with more than the maximum size. > > I suggest to restore the original behavior that initializes with an initialCapacity of 1 in most cases. That was true when before JDK-8210985, the property javax.net.ssl.sessionCacheSize was not set at all. > In case it was set, the initial size would have been like now, (javax.net.ssl.sessionCacheSize / 0.75f) + 1, which still seems strange. No regression test. ------------- Changes requested by xuelei (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/937 From xuelei at openjdk.java.net Thu Oct 29 17:16:47 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Thu, 29 Oct 2020 17:16:47 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 17:06:11 GMT, Volker Simonis wrote: >> It seems that there exists a memory/performance regression that was introduced with JDK-8210985: Update the default SSL session cache size to 20480. >> >> The idea to limit the maixmum SSL session cache size by itself is good. Unfortunately, as per the current implementation of sun.security.util.MemoryCache, it also modifies the initial size of the LinkedHashMap that is backing the cache to initialize with more than the maximum size. >> >> I suggest to restore the original behavior that initializes with an initialCapacity of 1 in most cases. That was true when before JDK-8210985, the property javax.net.ssl.sessionCacheSize was not set at all. >> In case it was set, the initial size would have been like now, (javax.net.ssl.sessionCacheSize / 0.75f) + 1, which still seems strange. > > Marked as reviewed by simonis (Reviewer). > Benchmarking is probably hard because we don't know the average occupancy of the map. > I agreed. No matter what the default value is, it will not fit perfectly in all situations. The value 1 may be fit for small workload applications, but not good for big workload applications. Applications could use the size setting APIs for the tuning. For this update, I think the impact for various workload may be limited/acceptable, but I'm not very sure of it. Benchmarking data with various workload would help us for a better sense. ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From simonis at openjdk.java.net Thu Oct 29 17:46:43 2020 From: simonis at openjdk.java.net (Volker Simonis) Date: Thu, 29 Oct 2020 17:46:43 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 17:14:17 GMT, Xue-Lei Andrew Fan wrote: >> It seems that there exists a memory/performance regression that was introduced with JDK-8210985: Update the default SSL session cache size to 20480. >> >> The idea to limit the maixmum SSL session cache size by itself is good. Unfortunately, as per the current implementation of sun.security.util.MemoryCache, it also modifies the initial size of the LinkedHashMap that is backing the cache to initialize with more than the maximum size. >> >> I suggest to restore the original behavior that initializes with an initialCapacity of 1 in most cases. That was true when before JDK-8210985, the property javax.net.ssl.sessionCacheSize was not set at all. >> In case it was set, the initial size would have been like now, (javax.net.ssl.sessionCacheSize / 0.75f) + 1, which still seems strange. > > No regression test. > > Benchmarking is probably hard because we don't know the average occupancy of the map. > > I agreed. No matter what the default value is, it will not fit perfectly in all situations. The value 1 may be fit for small workload applications, but not good for big workload applications. Applications could use the size setting APIs for the tuning. For this update, I think the impact for various workload may be limited/acceptable, but I'm not very sure of it. Benchmarking data with various workload would help us for a better sense. But we did run with `1` for quite a long time without somebody complaining :) ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From xuelei at openjdk.java.net Thu Oct 29 18:05:43 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Thu, 29 Oct 2020 18:05:43 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 17:43:43 GMT, Volker Simonis wrote: > > > Benchmarking is probably hard because we don't know the average occupancy of the map. > > > > > > I agreed. No matter what the default value is, it will not fit perfectly in all situations. The value 1 may be fit for small workload applications, but not good for big workload applications. Applications could use the size setting APIs for the tuning. For this update, I think the impact for various workload may be limited/acceptable, but I'm not very sure of it. Benchmarking data with various workload would help us for a better sense. > > But we did run with `1` for quite a long time without somebody complaining :) Yes, I think it is a safe update and looks good to me. I believe the impact should be minimal. But normally, I would like to check with a test for sure. If no regression test, an explain with noreg tag may be needed. External testing, like a confirmation of no performance regression any longer in an existing application, is fine. I don't want to block this integration, please go ahead if you are confident with it. ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From mullan at openjdk.java.net Thu Oct 29 18:11:49 2020 From: mullan at openjdk.java.net (Sean Mullan) Date: Thu, 29 Oct 2020 18:11:49 GMT Subject: RFR: 8255536: Remove the directsign property and option In-Reply-To: References: Message-ID: On Wed, 28 Oct 2020 20:56:31 GMT, Weijun Wang wrote: > I regret adding the `directsign` property/option to JarSigner/jarsigner. See the bug for details. Looks good. ------------- Marked as reviewed by mullan (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/915 From weijun at openjdk.java.net Thu Oct 29 18:37:06 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Thu, 29 Oct 2020 18:37:06 GMT Subject: RFR: 8255494: PKCS7 should use digest algorithm to verify the signature [v2] In-Reply-To: References: Message-ID: <_1vRWcaoLz_7O7m0XwHDrr3tOzx5AxrNpaXXW53HjcA=.4e038d02-0cb2-4731-a5ca-9aa411b5577c@github.com> > This is a regression made by [JDK-8242068](https://bugs.openjdk.java.net/browse/JDK-8242068). When the digest algorithm is not the same as the hash part of the signature algorithm, we used to combine the digest algorithm with the key part of the signature algorithm into a new signature algorithm and use it when generating a signature. The previous code change uses the signature algorithm in the SignerInfo directly. This bugfix will revert to the old behavior. Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: more comment to the test, and full DER encoding ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/916/files - new: https://git.openjdk.java.net/jdk/pull/916/files/bc354142..19aa3f4d Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=916&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=916&range=00-01 Stats: 9 lines in 1 file changed: 3 ins; 0 del; 6 mod Patch: https://git.openjdk.java.net/jdk/pull/916.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/916/head:pull/916 PR: https://git.openjdk.java.net/jdk/pull/916 From hchao at openjdk.java.net Thu Oct 29 19:00:43 2020 From: hchao at openjdk.java.net (Hai-May Chao) Date: Thu, 29 Oct 2020 19:00:43 GMT Subject: RFR: 8255494: PKCS7 should use digest algorithm to verify the signature In-Reply-To: References: Message-ID: On Wed, 28 Oct 2020 21:01:44 GMT, Weijun Wang wrote: > This is a regression made by [JDK-8242068](https://bugs.openjdk.java.net/browse/JDK-8242068). When the digest algorithm is not the same as the hash part of the signature algorithm, we used to combine the digest algorithm with the key part of the signature algorithm into a new signature algorithm and use it when generating a signature. The previous code change uses the signature algorithm in the SignerInfo directly. This bugfix will revert to the old behavior. Looks good! ------------- PR: https://git.openjdk.java.net/jdk/pull/916 From clanger at openjdk.java.net Thu Oct 29 23:24:47 2020 From: clanger at openjdk.java.net (Christoph Langer) Date: Thu, 29 Oct 2020 23:24:47 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 18:02:47 GMT, Xue-Lei Andrew Fan wrote: >>> > Benchmarking is probably hard because we don't know the average occupancy of the map. >>> >>> I agreed. No matter what the default value is, it will not fit perfectly in all situations. The value 1 may be fit for small workload applications, but not good for big workload applications. Applications could use the size setting APIs for the tuning. For this update, I think the impact for various workload may be limited/acceptable, but I'm not very sure of it. Benchmarking data with various workload would help us for a better sense. >> >> But we did run with `1` for quite a long time without somebody complaining :) > >> > > Benchmarking is probably hard because we don't know the average occupancy of the map. >> > >> > >> > I agreed. No matter what the default value is, it will not fit perfectly in all situations. The value 1 may be fit for small workload applications, but not good for big workload applications. Applications could use the size setting APIs for the tuning. For this update, I think the impact for various workload may be limited/acceptable, but I'm not very sure of it. Benchmarking data with various workload would help us for a better sense. >> >> But we did run with `1` for quite a long time without somebody complaining :) > > Yes, I think it is a safe update and looks good to me. I believe the impact should be minimal. But normally, I would like to check with a test for sure. If no regression test, an explain with noreg tag may be needed. External testing, like a confirmation of no performance regression any longer in an existing application, is fine. > > I don't want to block this integration, please go ahead if you are confident with it. I added the noreg-hard label to the JBS bug. I'll wait for any further input til monday, before integration. @XueleiFan would be nice if you could approve then, too :) ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From xuelei at openjdk.java.net Fri Oct 30 04:35:45 2020 From: xuelei at openjdk.java.net (Xue-Lei Andrew Fan) Date: Fri, 30 Oct 2020 04:35:45 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: <4b11RYB_Iq_DR1kqIJBomreHimvMNLZdyWrTDYrp0Gs=.8e2ee1db-81c4-49df-9bda-ee422d86430f@github.com> On Thu, 29 Oct 2020 15:11:09 GMT, Christoph Langer wrote: > It seems that there exists a memory/performance regression that was introduced with JDK-8210985: Update the default SSL session cache size to 20480. > > The idea to limit the maixmum SSL session cache size by itself is good. Unfortunately, as per the current implementation of sun.security.util.MemoryCache, it also modifies the initial size of the LinkedHashMap that is backing the cache to initialize with more than the maximum size. > > I suggest to restore the original behavior that initializes with an initialCapacity of 1 in most cases. That was true when before JDK-8210985, the property javax.net.ssl.sessionCacheSize was not set at all. > In case it was set, the initial size would have been like now, (javax.net.ssl.sessionCacheSize / 0.75f) + 1, which still seems strange. I had a benchmarking with one client and one server for full handshaking, with JMH. I did not see throughput regression with this benchmarking, and the improvement is not visible although. The benchmarking itself may be limited as it is trying to use a large volume of connections. ------------- Marked as reviewed by xuelei (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/937 From aph at openjdk.java.net Fri Oct 30 10:36:43 2020 From: aph at openjdk.java.net (Andrew Haley) Date: Fri, 30 Oct 2020 10:36:43 GMT Subject: RFR: JDK-8255603: Memory/Performance regression after JDK-8210985 In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 15:11:09 GMT, Christoph Langer wrote: > It seems that there exists a memory/performance regression that was introduced with JDK-8210985: Update the default SSL session cache size to 20480. > > The idea to limit the maixmum SSL session cache size by itself is good. Unfortunately, as per the current implementation of sun.security.util.MemoryCache, it also modifies the initial size of the LinkedHashMap that is backing the cache to initialize with more than the maximum size. > > I suggest to restore the original behavior that initializes with an initialCapacity of 1 in most cases. That was true when before JDK-8210985, the property javax.net.ssl.sessionCacheSize was not set at all. > In case it was set, the initial size would have been like now, (javax.net.ssl.sessionCacheSize / 0.75f) + 1, which still seems strange. Marked as reviewed by aph (Reviewer). src/java.base/share/classes/sun/security/util/Cache.java line 268: > 266: this.queue = null; > 267: > 268: cacheMap = new LinkedHashMap<>(1, LOAD_FACTOR, true); This looks right. The idea of scaling the initial cache size to the maximum size seems obviously to be wrong. ------------- PR: https://git.openjdk.java.net/jdk/pull/937 From mcimadamore at openjdk.java.net Fri Oct 30 11:40:58 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Fri, 30 Oct 2020 11:40:58 GMT Subject: RFR: 8254162: Implementation of Foreign-Memory Access API (Third Incubator) [v19] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the third incubation round of the foreign memory access API incubation (see JEP 393 [1]). This iteration focus on improving the usability of the API in 3 main ways: > > * first, by providing a way to obtain truly *shared* segments, which can be accessed and closed concurrently from multiple threads > * second, by providing a way to register a memory segment against a `Cleaner`, so as to have some (optional) guarantee that the memory will be deallocated, eventually > * third, by not requiring users to dive deep into var handles when they first pick up the API; a new `MemoryAccess` class has been added, which defines several useful dereference routines; these are really just thin wrappers around memory access var handles, but they make the barrier of entry for using this API somewhat lower. > > A big conceptual shift that comes with this API refresh is that the role of `MemorySegment` and `MemoryAddress` is not the same as it used to be; it used to be the case that a memory address could (sometimes, not always) have a back link to the memory segment which originated it; additionally, memory access var handles used `MemoryAddress` as a basic unit of dereference. > > This has all changed as per this API refresh; now a `MemoryAddress` is just a dumb carrier which wraps a pair of object/long addressing coordinates; `MemorySegment` has become the star of the show, as far as dereferencing memory is concerned. You cannot dereference memory if you don't have a segment. This improves usability in a number of ways - first, it is a lot easier to wrap native addresses (`long`, essentially) into a `MemoryAddress`; secondly, it is crystal clear what a client has to do in order to dereference memory: if a client has a segment, it can use that; otherwise, if the client only has an address, it will have to create a segment *unsafely* (this can be done by calling `MemoryAddress::asSegmentRestricted`). > > A list of the API, implementation and test changes is provided below. If you have any questions, or need more detailed explanations, I (and the rest of the Panama team) will be happy to point at existing discussions, and/or to provide the feedback required. > > A big thank to Erik Osterlund, Vladimir Ivanov and David Holmes, without whom the work on shared memory segment would not have been possible; also I'd like to thank Paul Sandoz, whose insights on API design have been very helpful in this journey. > > Thanks > Maurizio > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff: > > http://cr.openjdk.java.net/~mcimadamore/8254162_v1/specdiff/jdk/incubator/foreign/package-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254163 > > > > ### API Changes > > * `MemorySegment` > * drop factory for restricted segment (this has been moved to `MemoryAddress`, see below) > * added a no-arg factory for a native restricted segment representing entire native heap > * rename `withOwnerThread` to `handoff` > * add new `share` method, to create shared segments > * add new `registerCleaner` method, to register a segment against a cleaner > * add more helpers to create arrays from a segment e.g. `toIntArray` > * add some `asSlice` overloads (to make up for the fact that now segments are more frequently used as cursors) > * rename `baseAddress` to `address` (so that `MemorySegment` can implement `Addressable`) > * `MemoryAddress` > * drop `segment` accessor > * drop `rebase` method and replace it with `segmentOffset` which returns the offset (a `long`) of this address relative to a given segment > * `MemoryAccess` > * New class supporting several static dereference helpers; the helpers are organized by carrier and access mode, where a carrier is one of the usual suspect (a Java primitive, minus `boolean`); the access mode can be simple (e.g. access base address of given segment), or indexed, in which case the accessor takes a segment and either a low-level byte offset,or a high level logical index. The classification is reflected in the naming scheme (e.g. `getByte` vs. `getByteAtOffset` vs `getByteAtIndex`). > * `MemoryHandles` > * drop `withOffset` combinator > * drop `withStride` combinator > * the basic memory access handle factory now returns a var handle which takes a `MemorySegment` and a `long` - from which it is easy to derive all the other handles using plain var handle combinators. > * `Addressable` > * This is a new interface which is attached to entities which can be projected to a `MemoryAddress`. For now, both `MemoryAddress` and `MemorySegment` implement it; we have plans, with JEP 389 [2] to add more implementations. Clients can largely ignore this interface, which comes in really handy when defining native bindings with tools like `jextract`. > * `MemoryLayouts` > * A new layout, for machine addresses, has been added to the mix. > > > > ### Implementation changes > > There are two main things to discuss here: support for shared segments, and the general simplification of the memory access var handle support. > > #### Shared segments > > The support for shared segments cuts in pretty deep in the VM. Support for shared segments is notoriously hard to achieve, at least in a way that guarantees optimal access performances. This is caused by the fact that, if a segment is shared, it would be possible for a thread to close it while another is accessing it. > > After considering several options (see [3]), we zeroed onto an approach which is inspired by an happy idea that Andrew Haley had (and that he reminded me of at this year OpenJDK committer workshop - thanks!). The idea is that if we could *freeze* the world (e.g. with a GC pause), while a segment is closed, we could then prevent segments from being accessed concurrently to a close operation. For this to work, it is crucial that no GC safepoints can occur between a segment liveness check and the access itself (otherwise it would be possible for the accessing thread to stop just right before an unsafe call). It also relies on the fact that hotspot/C2 should not be able to propagate loads across safepoints. > > Sadly, none of these conditions seems to be valid in the current implementation, so we needed to resort to a bit of creativity. First, we noted that, if we could mark so called *scoped* method with an annotation, it would be very simply to check as to whether a thread was in the middle of a scoped method when we stopped the world for a close operation (btw, instead of stopping the world, we do a much more efficient, thread-local polling, thanks to JEP 312 [4]). > > The question is, then, once we detect that a thread is accessing the very segment we're about to close, what should happen? We first experimented with a solution which would install an *asynchronous* exception on the accessing thread, thus making it fail. This solution has some desirable properties, in that a `close` operation always succeeds. Unfortunately the machinery for async exceptions is a bit fragile (e.g. not all the code in hotspot checks for async exceptions); to minimize risks, we decided to revert to a simpler strategy, where `close` might fail when it finds that another thread is accessing the segment being closed. > > As written in the javadoc, this doesn't mean that clients should just catch and try again; an exception on `close` is a bug in the user code, likely arising from lack of synchronization, and should be treated as such. > > In terms of gritty implementation, we needed to centralize memory access routines in a single place, so that we could have a set of routines closely mimicking the primitives exposed by `Unsafe` but which, in addition, also provided a liveness check. This way we could mark all these routines with the special `@Scoped` annotation, which tells the VM that something important is going on. > > To achieve this, we created a new (autogenerated) class, called `ScopedMemoryAccess`. This class contains all the main memory access primitives (including bulk access, like `copyMemory`, or `setMemory`), and accepts, in addition to the access coordinates, also a scope object, which is tested before access. A reachability fence is also thrown in the mix to make sure that the scope is kept alive during access (which is important when registering segments against cleaners). > > Of course, to make memory access safe, memory access var handles, byte buffer var handles, and byte buffer API should use the new `ScopedMemoryAccess` class instead of unsafe, so that a liveness check can be triggered (in case a scope is present). > > `ScopedMemoryAccess` has a `closeScope` method, which initiates the thread-local handshakes, and returns `true` if the handshake completed successfully. > > The implementation of `MemoryScope` (now significantly simplified from what we had before), has two implementations, one for confined segments and one for shared segments; the main difference between the two is what happens when the scope is closed; a confined segment sets a boolean flag to false, and returns, whereas a shared segment goes into a `CLOSING` state, then starts the handshake, and then updates the state again, to either `CLOSED` or `ALIVE` depending on whether the handshake was successful or not. Note that when a shared segment is in the `CLOSING` state, `MemorySegment::isAlive` will still return `true`, while the liveness check upon memory access will fail. > > #### Memory access var handles overhaul > > The key realization here was that if all memory access var handles took a coordinate pair of `MemorySegment` and `long`, all other access types could be derived from this basic var handle form. > > This allowed us to remove the on-the-fly var handle generation, and to simply derive structural access var handles (such as those obtained by calling `MemoryLayout::varHandle`) using *plain* var handle combinators, so that e.g. additional offset is injected into a base memory access var handle. > > This also helped in simplifying the implementation by removing the special `withStride` and `withOffset` combinators, which previously needed low-level access on the innards of the memory access var handle. All that code is now gone. > > #### Test changes > > Not much to see here - most of the tests needed to be updated because of the API changes. Some were beefed up (like the array test, since now segments can be projected into many different kinds of arrays). A test has been added to test the `Cleaner` functionality, and another stress test has been added for shared segments (`TestHandshake`). Some of the microbenchmarks also needed some tweaks - and some of them were also updated to also test performance in the shared segment case. > > [1] - https://openjdk.java.net/jeps/393 > [2] - https://openjdk.java.net/jeps/389 > [3] - https://mail.openjdk.java.net/pipermail/panama-dev/2020-May/009004.html > [4] - https://openjdk.java.net/jeps/312 Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 25 commits: - Merge branch 'master' into 8254162 - Fix issues with derived buffers and IO operations - More 32-bit fixes for TestLayouts - * Add final to MappedByteBuffer::SCOPED_MEMORY_ACCESS field * Tweak TestLayouts to make it 32-bit friendly after recent MemoryLayouts tweaks - Remove TestMismatch from 32-bit problem list - Merge branch 'master' into 8254162 - Tweak javadoc for MemorySegment::mapFromPath Tweak alignment for long/double Java layouts on 32 bits platforms - Merge branch 'master' into 8254162 - Address review comment for scoped memory access makefile - Address CSR comments - ... and 15 more: https://git.openjdk.java.net/jdk/compare/e48016b1...bd400615 ------------- Changes: https://git.openjdk.java.net/jdk/pull/548/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=548&range=18 Stats: 7618 lines in 80 files changed: 4892 ins; 1537 del; 1189 mod Patch: https://git.openjdk.java.net/jdk/pull/548.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/548/head:pull/548 PR: https://git.openjdk.java.net/jdk/pull/548 From mcimadamore at openjdk.java.net Fri Oct 30 11:55:05 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Fri, 30 Oct 2020 11:55:05 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v13] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). > > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. > > Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. > > A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). > > Thanks > Maurizio > > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). > > ### Implementation changes > > The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). > > As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. > > Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. > > The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. > > This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. > > For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. > > A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. > > At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: > > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). > > Again, for more readings on the internals of the foreign linker support, please refer to [5]. > > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. > > Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. > > [1] - https://openjdk.java.net/jeps/389 > [2] - https://openjdk.java.net/jeps/393 > [3] - https://git.openjdk.java.net/jdk/pull/548 > [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md > [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 58 commits: - Merge branch '8254162' into 8254231_linker - Merge branch 'master' into 8254162 - Fix issues with derived buffers and IO operations - More 32-bit fixes for TestLayouts - * Add final to MappedByteBuffer::SCOPED_MEMORY_ACCESS field * Tweak TestLayouts to make it 32-bit friendly after recent MemoryLayouts tweaks - Remove TestMismatch from 32-bit problem list - Merge branch 'master' into 8254162 - Tweak javadoc for MemorySegment::mapFromPath Tweak alignment for long/double Java layouts on 32 bits platforms - Merge branch 'master' into 8254162 - Address review comment for scoped memory access makefile - ... and 48 more: https://git.openjdk.java.net/jdk/compare/e48016b1...4a2c2240 ------------- Changes: https://git.openjdk.java.net/jdk/pull/634/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=12 Stats: 75299 lines in 271 files changed: 72395 ins; 1615 del; 1289 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From mcimadamore at openjdk.java.net Fri Oct 30 12:16:02 2020 From: mcimadamore at openjdk.java.net (Maurizio Cimadamore) Date: Fri, 30 Oct 2020 12:16:02 GMT Subject: RFR: 8254231: Implementation of Foreign Linker API (Incubator) [v14] In-Reply-To: References: Message-ID: > This patch contains the changes associated with the first incubation round of the foreign linker access API incubation > (see JEP 389 [1]). This work is meant to sit on top of the foreign memory access support (see JEP 393 [2] and associated pull request [3]). > > The main goal of this API is to provide a way to call native functions from Java code without the need of intermediate JNI glue code. In order to do this, native calls are modeled through the MethodHandle API. I suggest reading the writeup [4] I put together few weeks ago, which illustrates what the foreign linker support is, and how it should be used by clients. > > Disclaimer: the pull request mechanism isn't great at managing *dependent* reviews. For this reasons, I'm attaching a webrev which contains only the differences between this PR and the memory access PR. I will be periodically uploading new webrevs, as new iterations come out, to try and make the life of reviewers as simple as possible. > > A big thank to Jorn Vernee and Vladimir Ivanov - they are the main architects of all the hotspot changes you see here, and without their help, the foreign linker support wouldn't be what it is today. As usual, a big thank to Paul Sandoz, who provided many insights (often by trying the bits first hand). > > Thanks > Maurizio > > Webrev: > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/webrev > > Javadoc: > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/javadoc/jdk/incubator/foreign/package-summary.html > > Specdiff (relative to [3]): > > http://cr.openjdk.java.net/~mcimadamore/8254231_v1/specdiff_delta/overview-summary.html > > CSR: > > https://bugs.openjdk.java.net/browse/JDK-8254232 > > > > ### API Changes > > The API changes are actually rather slim: > > * `LibraryLookup` > * This class allows clients to lookup symbols in native libraries; the interface is fairly simple; you can load a library by name, or absolute path, and then lookup symbols on that library. > * `FunctionDescriptor` > * This is an abstraction that is very similar, in spirit, to `MethodType`; it is, at its core, an aggregate of memory layouts for the function arguments/return type. A function descriptor is used to describe the signature of a native function. > * `CLinker` > * This is the real star of the show. A `CLinker` has two main methods: `downcallHandle` and `upcallStub`; the first takes a native symbol (as obtained from `LibraryLookup`), a `MethodType` and a `FunctionDescriptor` and returns a `MethodHandle` instance which can be used to call the target native symbol. The second takes an existing method handle, and a `FunctionDescriptor` and returns a new `MemorySegment` corresponding to a code stub allocated by the VM which acts as a trampoline from native code to the user-provided method handle. This is very useful for implementing upcalls. > * This class also contains the various layout constants that should be used by clients when describing native signatures (e.g. `C_LONG` and friends); these layouts contain additional ABI classfication information (in the form of layout attributes) which is used by the runtime to *infer* how Java arguments should be shuffled for the native call to take place. > * Finally, this class provides some helper functions e.g. so that clients can convert Java strings into C strings and back. > * `NativeScope` > * This is an helper class which allows clients to group together logically related allocations; that is, rather than allocating separate memory segments using separate *try-with-resource* constructs, a `NativeScope` allows clients to use a _single_ block, and allocate all the required segments there. This is not only an usability boost, but also a performance boost, since not all allocation requests will be turned into `malloc` calls. > * `MemorySegment` > * Only one method added here - namely `handoff(NativeScope)` which allows a segment to be transferred onto an existing native scope. > > ### Safety > > The foreign linker API is intrinsically unsafe; many things can go wrong when requesting a native method handle. For instance, the description of the native signature might be wrong (e.g. have too many arguments) - and the runtime has, in the general case, no way to detect such mismatches. For these reasons, obtaining a `CLinker` instance is a *restricted* operation, which can be enabled by specifying the usual JDK property `-Dforeign.restricted=permit` (as it's the case for other restricted method in the foreign memory API). > > ### Implementation changes > > The Java changes associated with `LibraryLookup` are relative straightforward; the only interesting thing to note here is that library loading does _not_ depend on class loaders, so `LibraryLookup` is not subject to the same restrictions which apply to JNI library loading (e.g. same library cannot be loaded by different classloaders). > > As for `NativeScope` the changes are again relatively straightforward; it is an API which sits neatly on top of the foreign meory access API, providing some kind of allocation service which shares the same underlying memory segment(s), and turns an allocation request into a segment slice, which is a much less expensive operation. `NativeScope` comes in two variants: there are native scopes for which the allocation size is known a priori, and native scopes which can grow - these two schemes are implemented by two separate subclasses of `AbstractNativeScopeImpl`. > > Of course the bulk of the changes are to support the `CLinker` downcall/upcall routines. These changes cut pretty deep into the JVM; I'll briefly summarize the goal of some of this changes - for further details, Jorn has put together a detailed writeup which explains the rationale behind the VM support, with some references to the code [5]. > > The main idea behind foreign linker is to infer, given a Java method type (expressed as a `MethodType` instance) and the description of the signature of a native function (expressed as a `FunctionDescriptor` instance) a _recipe_ that can be used to turn a Java call into the corresponding native call targeting the requested native function. > > This inference scheme can be defined in a pretty straightforward fashion by looking at the various ABI specifications (for instance, see [6] for the SysV ABI, which is the one used on Linux/Mac). The various `CallArranger` classes, of which we have a flavor for each supported platform, do exactly that kind of inference. > > For the inference process to work, we need to attach extra information to memory layouts; it is no longer sufficient to know e.g. that a layout is 32/64 bits - we need to know whether it is meant to represent a floating point value, or an integral value; this knowledge is required because floating points are passed in different registers by most ABIs. For this reason, `CLinker` offers a set of pre-baked, platform-dependent layout constants which contain the required classification attributes (e.g. a `Clinker.TypeKind` enum value). The runtime extracts this attribute, and performs classification accordingly. > > A native call is decomposed into a sequence of basic, primitive operations, called `Binding` (see the great javadoc on the `Binding.java` class for more info). There are many such bindings - for instance the `Move` binding is used to move a value into a specific machine register/stack slot. So, the main job of the various `CallingArranger` classes is to determine, given a Java `MethodType` and `FunctionDescriptor` what is the set of bindings associated with the downcall/upcall. > > At the heart of the foreign linker support is the `ProgrammableInvoker` class. This class effectively generates a `MethodHandle` which follows the steps described by the various bindings obtained by `CallArranger`. There are actually various strategies to interpret these bindings - listed below: > > * basic intepreted mode; in this mode, all bindings are interpreted using a stack-based machine written in Java (see `BindingInterpreter`), except for the `Move` bindings. For these bindings, the move is implemented by allocating a *buffer* (whose size is ABI specific) and by moving all the lowered values into positions within this buffer. The buffer is then passed to a piece of assembly code inside the VM which takes values from the buffer and moves them in their expected registers/stack slots (note that each position in the buffer corresponds to a different register). This is the most general invocation mode, the more "customizable" one, but also the slowest - since for every call there is some extra allocation which takes place. > > * specialized interpreted mode; same as before, but instead of interpreting the bindings with a stack-based interpreter, we generate a method handle chain which effectively interprets all the bindings (again, except `Move` ones). > > * intrinsified mode; this is typically used in combination with the specialized interpreted mode described above (although it can also be used with the Java-based binding interpreter). The goal here is to remove the buffer allocation and copy by introducing an additional JVM intrinsic. If a native call recipe is constant (e.g. the set of bindings is constant, which is probably the case if the native method handle is stored in a `static`, `final` field), then the VM can generate specialized assembly code which interprets the `Move` binding without the need to go for an intermediate buffer. This gives us back performances that are on par with JNI. > > For upcalls, the support is not (yet) as advanced, and only the basic interpreted mode is available there. We plan to add support for intrinsified modes there as well, which should considerably boost perfomances (probably well beyond what JNI can offer at the moment, since the upcall support in JNI is not very well optimized). > > Again, for more readings on the internals of the foreign linker support, please refer to [5]. > > #### Test changes > > Many new tests have been added to validate the foreign linker support; we have high level tests (see `StdLibTest`) which aim at testing the linker from the perspective of code that clients could write. But we also have deeper combinatorial tests (see `TestUpcall` and `TestDowncall`) which are meant to stress every corner of the ABI implementation. There are also some great tests (see the `callarranger` folder) which test the various `CallArranger`s for all the possible platforms; these tests adopt more of a white-box approach - that is, instead of treating the linker machinery as a black box and verify that the support works by checking that the native call returned the results we expected, these tests aims at checking that the set of bindings generated by the call arranger is correct. This also mean that we can test the classification logic for Windows, Mac and Linux regardless of the platform we're executing on. > > Some additional microbenchmarks have been added to compare the performances of downcall/upcall with JNI. > > [1] - https://openjdk.java.net/jeps/389 > [2] - https://openjdk.java.net/jeps/393 > [3] - https://git.openjdk.java.net/jdk/pull/548 > [4] - https://github.com/openjdk/panama-foreign/blob/foreign-jextract/doc/panama_ffi.md > [5] - http://cr.openjdk.java.net/~jvernee/docs/Foreign-abi%20downcall%20intrinsics%20technical%20description.html Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Fix typo in upcall helper for aarch64 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/634/files - new: https://git.openjdk.java.net/jdk/pull/634/files/4a2c2240..98718866 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=13 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=634&range=12-13 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/634.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/634/head:pull/634 PR: https://git.openjdk.java.net/jdk/pull/634 From mullan at openjdk.java.net Fri Oct 30 12:35:56 2020 From: mullan at openjdk.java.net (Sean Mullan) Date: Fri, 30 Oct 2020 12:35:56 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v3] In-Reply-To: References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: On Fri, 9 Oct 2020 01:33:38 GMT, Weijun Wang wrote: >> Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at https://bugs.openjdk.java.net/browse/JDK-8228481. > > Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: > > update README and exclude README Marked as reviewed by mullan (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Fri Oct 30 13:30:21 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 30 Oct 2020 13:30:21 GMT Subject: RFR: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms [v4] In-Reply-To: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: <2cpMwHqezg0PlXKECF1XE4XM4XlS7UDCcoRw7wGEDzg=.6c296cc2-3d18-4d89-9d6b-7007989463bd@github.com> > Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at https://bugs.openjdk.java.net/browse/JDK-8228481. Weijun Wang has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision: - simplify test - merge - update README and exclude README - change ic to 10000 - 8153005: Upgrade the default PKCS12 encryption/MAC algorithms ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/473/files - new: https://git.openjdk.java.net/jdk/pull/473/files/41be78aa..31a22fd4 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=473&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=473&range=02-03 Stats: 401475 lines in 1918 files changed: 365145 ins; 23887 del; 12443 mod Patch: https://git.openjdk.java.net/jdk/pull/473.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/473/head:pull/473 PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Fri Oct 30 13:30:22 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 30 Oct 2020 13:30:22 GMT Subject: Integrated: 8153005: Upgrade the default PKCS12 encryption/MAC algorithms In-Reply-To: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> References: <6k0tdplQ0jvii6Ikk-djEL9BGmjeEOpgouufuYgQdTc=.84452f42-b8c6-4e7c-ba29-514891f1cdcb@github.com> Message-ID: On Thu, 1 Oct 2020 20:02:34 GMT, Weijun Wang wrote: > Default algorithms are bumped to be based on PBES2 with AES-256 and SHA-256. Please also review the CSR at https://bugs.openjdk.java.net/browse/JDK-8228481. This pull request has now been integrated. Changeset: f77a6585 Author: Weijun Wang URL: https://git.openjdk.java.net/jdk/commit/f77a6585 Stats: 511 lines in 8 files changed: 172 ins; 113 del; 226 mod 8153005: Upgrade the default PKCS12 encryption/MAC algorithms Reviewed-by: mullan ------------- PR: https://git.openjdk.java.net/jdk/pull/473 From weijun at openjdk.java.net Fri Oct 30 13:33:55 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 30 Oct 2020 13:33:55 GMT Subject: Integrated: 8255536: Remove the directsign property and option In-Reply-To: References: Message-ID: <8COVKgwBr7BM6M0bsDoH1nVV4Ab6QU_PBh_7FSYfFu0=.e5c90ac4-80c2-4ab5-9084-d5fa2986fd1e@github.com> On Wed, 28 Oct 2020 20:56:31 GMT, Weijun Wang wrote: > I regret adding the `directsign` property/option to JarSigner/jarsigner. See the bug for details. This pull request has now been integrated. Changeset: a7563207 Author: Weijun Wang URL: https://git.openjdk.java.net/jdk/commit/a7563207 Stats: 184 lines in 7 files changed: 14 ins; 162 del; 8 mod 8255536: Remove the directsign property and option Reviewed-by: mullan ------------- PR: https://git.openjdk.java.net/jdk/pull/915 From weijun at openjdk.java.net Fri Oct 30 14:45:56 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Fri, 30 Oct 2020 14:45:56 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 03:18:33 GMT, Weijun Wang wrote: >> Changes look good. Only minor comments. > > Just curious, can the Java files be generated during the build process? The constants in PKCS11Exception are duplicated in PKCS11Constants. 0x00000000, vs public static final long CKR_OK = 0x00000000L; Is there any way to simplify it? ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From valeriep at openjdk.java.net Fri Oct 30 21:36:00 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 30 Oct 2020 21:36:00 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: <4EoLrlw-bXUO2d-nX5-bZifDCwVIpKNgWE6nctG4FBM=.a6618c7d-bbf3-4279-9f92-75401a8f4ccf@github.com> On Thu, 29 Oct 2020 02:06:06 GMT, Hai-May Chao wrote: >> Could someone please help review this PKCS#11 v3.0 header files update? >> >> Changes are straight-forward as below: >> 1) Updated pkcs11.h, pkcs11f.h, pkcs11t.h to v3.0 >> 2) Updated java side w/ the new constants definitions and name/error code mappings. >> >> For the native headers, it's a direct copy of the official v3.0 headers except that I have to remove the tab space, and trailing white spaces due to JDK code requirement. I verified the result using 'diff -w'. As for the java side, the edit is based on the diff of native headers. I also commented out some of the unused native identifiers at java side. >> >> I am adding the SHA-3 digests, signatures, and macs in a separate RFE and would need this one to be reviewed/integrated first. >> >> Thanks, >> Valerie > > src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/wrapper/Functions.java line 793: > >> 791: addMech(CKM_SHA3_512_RSA_PKCS_PSS, "CKM_SHA3_512_RSA_PKCS_PSS"); >> 792: addMech(CKM_SHA3_224_RSA_PKCS, "CKM_SHA3_224_RSA_PKCS"); >> 793: addMech(CKM_SHA3_224_RSA_PKCS_PSS, "CKM_SHA3_224_RSA_PKCS_PSS"); > > It appears that you're arranging the addMech(with CKM_xxx) based on the mechanism values. How about the code from #773 to #793, move it up? Yes, I will move it up and add a comment about overall ordering is based on PKCS11Constants class. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From valeriep at openjdk.java.net Fri Oct 30 21:42:59 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 30 Oct 2020 21:42:59 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 02:07:39 GMT, Hai-May Chao wrote: >> Could someone please help review this PKCS#11 v3.0 header files update? >> >> Changes are straight-forward as below: >> 1) Updated pkcs11.h, pkcs11f.h, pkcs11t.h to v3.0 >> 2) Updated java side w/ the new constants definitions and name/error code mappings. >> >> For the native headers, it's a direct copy of the official v3.0 headers except that I have to remove the tab space, and trailing white spaces due to JDK code requirement. I verified the result using 'diff -w'. As for the java side, the edit is based on the diff of native headers. I also commented out some of the unused native identifiers at java side. >> >> I am adding the SHA-3 digests, signatures, and macs in a separate RFE and would need this one to be reviewed/integrated first. >> >> Thanks, >> Valerie > > src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/wrapper/Functions.java line 1095: > >> 1093: addMech(CKM_SP800_108_FEEDBACK_KDF, "CKM_SP800_108_FEEDBACK_KDF"); >> 1094: addMech(CKM_SP800_108_DOUBLE_PIPELINE_KDF, >> 1095: "CKM_SP800_108_DOUBLE_PIPELINE_KDF"); > > same comment as above. Well, per the ordering in PKCS11Constants, these three lines are at the right place. Note that the ordering of CKM_ECDSA_SHA3_224 to CKM_EDDSA in pkcs11t.h is different from PKCS11Constants class, so I will add the comment about the general ordering following PKCS11Constants class and keep them here. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From valeriep at openjdk.java.net Fri Oct 30 21:46:57 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 30 Oct 2020 21:46:57 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 02:16:15 GMT, Hai-May Chao wrote: >> Could someone please help review this PKCS#11 v3.0 header files update? >> >> Changes are straight-forward as below: >> 1) Updated pkcs11.h, pkcs11f.h, pkcs11t.h to v3.0 >> 2) Updated java side w/ the new constants definitions and name/error code mappings. >> >> For the native headers, it's a direct copy of the official v3.0 headers except that I have to remove the tab space, and trailing white spaces due to JDK code requirement. I verified the result using 'diff -w'. As for the java side, the edit is based on the diff of native headers. I also commented out some of the unused native identifiers at java side. >> >> I am adding the SHA-3 digests, signatures, and macs in a separate RFE and would need this one to be reviewed/integrated first. >> >> Thanks, >> Valerie > > src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/wrapper/PKCS11Constants.java line 987: > >> 985: public static final long CKM_SP800_108_FEEDBACK_KDF = 0x000003adL; >> 986: public static final long CKM_SP800_108_DOUBLE_PIPELINE_KDF = 0x000003aeL; >> 987: > > Same comment. These three are just by themselves, so unless you feel strongly about, I prefer just leave them here which matches the ordering of pkcs11t.h, i.e. right before the CKM_VENDOR_DEFINED line. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From valeriep at openjdk.java.net Fri Oct 30 21:50:58 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 30 Oct 2020 21:50:58 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: <8MpFlDgrQVFbJS_A6VW2YEGaL5Zy5WpiM4HMjC2kR34=.1fd69960-8d95-45b0-9be6-afa35baffc0f@github.com> On Thu, 29 Oct 2020 03:18:33 GMT, Weijun Wang wrote: > > > Just curious, can the Java files be generated during the build process? Hmm, maybe, by the java files, do you just mean PKCS11Constants class or more? I am not familiar with how to generate Java files during the build process, can you share some pointers? I can look into them for possible future enhancement. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From valeriep at openjdk.java.net Fri Oct 30 21:57:54 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 30 Oct 2020 21:57:54 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Fri, 30 Oct 2020 14:43:20 GMT, Weijun Wang wrote: > > > The constants in PKCS11Exception are duplicated in PKCS11Constants. > > ``` > 0x00000000, > ``` > > vs > > ``` > public static final long CKR_OK = 0x00000000L; > ``` > > Is there any way to simplify it? One defines the value, the other defines the displayed String. I agree that the way it's currently done is a bit outdated and may be error prone. We can re-visit this after finishing other pending PKCS11 RFEs.. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From valeriep at openjdk.java.net Fri Oct 30 23:02:58 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 30 Oct 2020 23:02:58 GMT Subject: RFR: 8255494: PKCS7 should use digest algorithm to verify the signature [v2] In-Reply-To: <_1vRWcaoLz_7O7m0XwHDrr3tOzx5AxrNpaXXW53HjcA=.4e038d02-0cb2-4731-a5ca-9aa411b5577c@github.com> References: <_1vRWcaoLz_7O7m0XwHDrr3tOzx5AxrNpaXXW53HjcA=.4e038d02-0cb2-4731-a5ca-9aa411b5577c@github.com> Message-ID: <8J-3cC_4Qoj2ct7FrNUETuoxDA-VnqFz_hoG5Bpmlfw=.74298421-a444-4cb8-8e6d-2e9374604cbd@github.com> On Thu, 29 Oct 2020 18:37:06 GMT, Weijun Wang wrote: >> This is a regression made by [JDK-8242068](https://bugs.openjdk.java.net/browse/JDK-8242068). When the digest algorithm is not the same as the hash part of the signature algorithm, we used to combine the digest algorithm with the key part of the signature algorithm into a new signature algorithm and use it when generating a signature. The previous code change uses the signature algorithm in the SignerInfo directly. This bugfix will revert to the old behavior. > > Weijun Wang has updated the pull request incrementally with one additional commit since the last revision: > > more comment to the test, and full DER encoding Marked as reviewed by valeriep (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/916 From valeriep at openjdk.java.net Fri Oct 30 23:02:59 2020 From: valeriep at openjdk.java.net (Valerie Peng) Date: Fri, 30 Oct 2020 23:02:59 GMT Subject: RFR: 8255494: PKCS7 should use digest algorithm to verify the signature In-Reply-To: References: Message-ID: On Thu, 29 Oct 2020 18:57:45 GMT, Hai-May Chao wrote: >> This is a regression made by [JDK-8242068](https://bugs.openjdk.java.net/browse/JDK-8242068). When the digest algorithm is not the same as the hash part of the signature algorithm, we used to combine the digest algorithm with the key part of the signature algorithm into a new signature algorithm and use it when generating a signature. The previous code change uses the signature algorithm in the SignerInfo directly. This bugfix will revert to the old behavior. > > Looks good! Looks good to me. ------------- PR: https://git.openjdk.java.net/jdk/pull/916 From hchao at openjdk.java.net Fri Oct 30 23:46:57 2020 From: hchao at openjdk.java.net (Hai-May Chao) Date: Fri, 30 Oct 2020 23:46:57 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Fri, 30 Oct 2020 21:39:42 GMT, Valerie Peng wrote: >> src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/wrapper/Functions.java line 1095: >> >>> 1093: addMech(CKM_SP800_108_FEEDBACK_KDF, "CKM_SP800_108_FEEDBACK_KDF"); >>> 1094: addMech(CKM_SP800_108_DOUBLE_PIPELINE_KDF, >>> 1095: "CKM_SP800_108_DOUBLE_PIPELINE_KDF"); >> >> same comment as above. > > Well, per the ordering in PKCS11Constants, these three lines are at the right place. Note that the ordering of CKM_ECDSA_SHA3_224 to CKM_EDDSA in pkcs11t.h is different from PKCS11Constants class, so I will add the comment about the general ordering following PKCS11Constants class and keep them here. Sounds good that a comment will be added. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From hchao at openjdk.java.net Sat Oct 31 00:08:59 2020 From: hchao at openjdk.java.net (Hai-May Chao) Date: Sat, 31 Oct 2020 00:08:59 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Fri, 30 Oct 2020 21:44:00 GMT, Valerie Peng wrote: >> src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/wrapper/PKCS11Constants.java line 987: >> >>> 985: public static final long CKM_SP800_108_FEEDBACK_KDF = 0x000003adL; >>> 986: public static final long CKM_SP800_108_DOUBLE_PIPELINE_KDF = 0x000003aeL; >>> 987: >> >> Same comment. > > These three are just by themselves, so unless you feel strongly about, I prefer just leave them here which matches the ordering of pkcs11t.h, i.e. right before the CKM_VENDOR_DEFINED line. Just thought they could be moved like CKM_ECDSA_SHA3_224 to CKM_EDDSA (which are not matching to the ordering of pkcs11t.h), so we keep the ordering in PKCS11Constants. It's fine if you'd prefer as is. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From weijun at openjdk.java.net Sat Oct 31 03:26:59 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Sat, 31 Oct 2020 03:26:59 GMT Subject: Integrated: 8255494: PKCS7 should use digest algorithm to verify the signature In-Reply-To: References: Message-ID: On Wed, 28 Oct 2020 21:01:44 GMT, Weijun Wang wrote: > This is a regression made by [JDK-8242068](https://bugs.openjdk.java.net/browse/JDK-8242068). When the digest algorithm is not the same as the hash part of the signature algorithm, we used to combine the digest algorithm with the key part of the signature algorithm into a new signature algorithm and use it when generating a signature. The previous code change uses the signature algorithm in the SignerInfo directly. This bugfix will revert to the old behavior. This pull request has now been integrated. Changeset: 80380d51 Author: Weijun Wang URL: https://git.openjdk.java.net/jdk/commit/80380d51 Stats: 129 lines in 3 files changed: 116 ins; 5 del; 8 mod 8255494: PKCS7 should use digest algorithm to verify the signature Reviewed-by: valeriep ------------- PR: https://git.openjdk.java.net/jdk/pull/916 From weijun at openjdk.java.net Sat Oct 31 03:30:54 2020 From: weijun at openjdk.java.net (Weijun Wang) Date: Sat, 31 Oct 2020 03:30:54 GMT Subject: RFR: 8244154: Update SunPKCS11 provider with PKCS11 v3.0 header files In-Reply-To: References: Message-ID: On Fri, 30 Oct 2020 21:55:10 GMT, Valerie Peng wrote: > > The constants in PKCS11Exception are duplicated in PKCS11Constants. > > ``` > > 0x00000000, > > ``` > > > > > > vs > > ``` > > public static final long CKR_OK = 0x00000000L; > > ``` > > > > > > Is there any way to simplify it? > > One defines the value, the other defines the displayed String. I agree that the way it's currently done is a bit outdated and may be error prone. We can re-visit this after finishing other pending PKCS11 RFEs.. Since the value is already defined PKCS11Constants, we can use it in PKCSException and does not need to write 0x00000000 anymore. ------------- PR: https://git.openjdk.java.net/jdk/pull/917 From djelinski1 at gmail.com Sat Oct 31 12:45:05 2020 From: djelinski1 at gmail.com (=?UTF-8?Q?Daniel_Jeli=C5=84ski?=) Date: Sat, 31 Oct 2020 13:45:05 +0100 Subject: ECC Key Usage ignored In-Reply-To: References: Message-ID: Verified that this behavior is still observed with JDK 16 EA 22. Client side code responsible for server certificate validation is located in EndEntityChecker.checkTLSServer [1]. That code is not executed when the certificate is trusted [2]. I believe this is a bug - I wouldn't choose to accept a server certificate when that certificate was only meant for signing other certificates, for example. Server side code responsible for cipher selection may need some refactoring; currently the server selects the first cipher that passes validation checks and for which keyManager.chooseServerAlias(keyType,...) returns a key. The key manager does not have information about the intended key usage (keyType is "EC" for both ECDH and ECDSA, and we don't know which cipher is being checked), so it returns the same certificate when querying for ECDH and ECDSA. Fortunately ECDH is not popular, and most clients won't even try to negotiate it. Still, I believe this is a bug that should be addressed. As far as I could tell, there were no bugs reported for this in the bug system. Should I report them there? If someone is interested in the server code, here's the interesting portion of stack trace: chooseServerAlias:260, SunX509KeyManagerImpl (sun.security.ssl) createServerPossession:288, X509Authentication$X509PossessionGenerator (sun.security.ssl) createPossession:214, X509Authentication$X509PossessionGenerator (sun.security.ssl) createPossession:90, X509Authentication (sun.security.ssl) createPossessions:51, SSLKeyExchange (sun.security.ssl) chooseCipherSuite:443, ServerHello$T12ServerHelloProducer (sun.security.ssl) [1] http://hg.openjdk.java.net/jdk/jdk/file/ee1d592a9f53/src/java.base/share/classes/sun/security/validator/EndEntityChecker.java#l276 [2] http://hg.openjdk.java.net/jdk/jdk/file/ee1d592a9f53/src/java.base/share/classes/sun/security/validator/Validator.java#l267 wt., 27 pa? 2020 o 18:44 Daniel Jeli?ski napisa?(a): > Hi all, > > TL;DR: both SSL server and client ignore KeyUsage certificate extension > when determining the list of available cipher suites. They shouldn't; > KeyUsage is the only differentiator between ECDH and ECDSA certificates. > > Long version: > I'm experimenting with ECC certificates on my Jetty server; when I created > an ECC certificate and tested the server with nmap, I found that both ECDSA > and ECDH cipher suites are enabled. I don't want ECDH ciphers, but I don't > want to add explicit excludes either. > > Reading into NIST recommendations [1] I found that ECDSA certificates > should define KeyUsage extension with value digitalSignature, vs ECDH which > should use keyAgreement value. > I experimented with both combinations of KeyValue, both resulted in the > same set of ciphers being offered by the server. The client doesn't seem to > care about KeyUsage either - it accepts connections even when the selected > cipher doesn't match KeyUsage. > > Chrome browser doesn't support ECDH ciphers, but it does support ECDSA. > When connecting to a Java server using ECDH certificate, it displays the > error "ERR_SSL_KEY_USAGE_INCOMPATIBLE"; the server offers an ECDSA cipher > suite, which is rejected by the browser. > > The issue was already reported by Bernd Eckenfels here [2], but as far as > I can tell, it is not addressed yet; I was able to reproduce it using > slightly modified code of this gist [3]. Certificates were generated using > keytool commands: > > ECDSA: > keytool -genkeypair -alias ec -keyalg EC -keysize 256 -sigalg > SHA256withECDSA -validity 365 -dname > "CN=localhost,OU=Unknown,O=Unknown,L=Unknown,S=Unknown,C=Unknown" > -storetype JKS -keystore ectest.jks -storepass 123456 -ext > KeyUsage:c=digitalSignature,keyCertSign > > ECDH: > keytool -genkeypair -alias ec -keyalg EC -keysize 256 -sigalg > SHA256withECDSA -validity 365 -dname > "CN=localhost,OU=Unknown,O=Unknown,L=Unknown,S=Unknown,C=Unknown" > -storetype JKS -keystore ectest.jks -storepass 123456 -ext > KeyUsage:c=keyAgreement,keyCertSign > > I'm not sure if keyCertSign is required on self-signed certificates, added > it just in case. > > Tested on OpenJDK 11.0.6. > > Regards, > Daniel Jeli?ski > > > [1] > https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-52r2.pdf > [2] > http://mail.openjdk.java.net/pipermail/security-dev/2017-May/015902.html > [3] https://gist.github.com/djelinski/b4543a3eb7ea66306044c08b41bba00f > -------------- next part -------------- An HTML attachment was scrubbed... URL: From XUELEI.FAN at ORACLE.COM Sat Oct 31 16:23:42 2020 From: XUELEI.FAN at ORACLE.COM (Xue-Lei Fan) Date: Sat, 31 Oct 2020 09:23:42 -0700 Subject: ECC Key Usage ignored In-Reply-To: References: Message-ID: Hi Daniel, Would you mind file a bug for the tracking? Xuelei > On Oct 31, 2020, at 5:45 AM, Daniel Jeli?ski wrote: > > Verified that this behavior is still observed with JDK 16 EA 22. > > Client side code responsible for server certificate validation is located in EndEntityChecker.checkTLSServer [1]. That code is not executed when the certificate is trusted [2]. I believe this is a bug - I wouldn't choose to accept a server certificate when that certificate was only meant for signing other certificates, for example. > > Server side code responsible for cipher selection may need some refactoring; currently the server selects the first cipher that passes validation checks and for which keyManager.chooseServerAlias(keyType,...) returns a key. The key manager does not have information about the intended key usage (keyType is "EC" for both ECDH and ECDSA, and we don't know which cipher is being checked), so it returns the same certificate when querying for ECDH and ECDSA. > > Fortunately ECDH is not popular, and most clients won't even try to negotiate it. Still, I believe this is a bug that should be addressed. > > As far as I could tell, there were no bugs reported for this in the bug system. Should I report them there? > > If someone is interested in the server code, here's the interesting portion of stack trace: > chooseServerAlias:260, SunX509KeyManagerImpl (sun.security.ssl) > createServerPossession:288, X509Authentication$X509PossessionGenerator (sun.security.ssl) > createPossession:214, X509Authentication$X509PossessionGenerator (sun.security.ssl) > createPossession:90, X509Authentication (sun.security.ssl) > createPossessions:51, SSLKeyExchange (sun.security.ssl) > chooseCipherSuite:443, ServerHello$T12ServerHelloProducer (sun.security.ssl) > > [1] http://hg.openjdk.java.net/jdk/jdk/file/ee1d592a9f53/src/java.base/share/classes/sun/security/validator/EndEntityChecker.java#l276 > [2] http://hg.openjdk.java.net/jdk/jdk/file/ee1d592a9f53/src/java.base/share/classes/sun/security/validator/Validator.java#l267 > wt., 27 pa? 2020 o 18:44 Daniel Jeli?ski > napisa?(a): > Hi all, > > TL;DR: both SSL server and client ignore KeyUsage certificate extension when determining the list of available cipher suites. They shouldn't; KeyUsage is the only differentiator between ECDH and ECDSA certificates. > > Long version: > I'm experimenting with ECC certificates on my Jetty server; when I created an ECC certificate and tested the server with nmap, I found that both ECDSA and ECDH cipher suites are enabled. I don't want ECDH ciphers, but I don't want to add explicit excludes either. > > Reading into NIST recommendations [1] I found that ECDSA certificates should define KeyUsage extension with value digitalSignature, vs ECDH which should use keyAgreement value. > I experimented with both combinations of KeyValue, both resulted in the same set of ciphers being offered by the server. The client doesn't seem to care about KeyUsage either - it accepts connections even when the selected cipher doesn't match KeyUsage. > > Chrome browser doesn't support ECDH ciphers, but it does support ECDSA. When connecting to a Java server using ECDH certificate, it displays the error "ERR_SSL_KEY_USAGE_INCOMPATIBLE"; the server offers an ECDSA cipher suite, which is rejected by the browser. > > The issue was already reported by Bernd Eckenfels here [2], but as far as I can tell, it is not addressed yet; I was able to reproduce it using slightly modified code of this gist [3]. Certificates were generated using keytool commands: > > ECDSA: > keytool -genkeypair -alias ec -keyalg EC -keysize 256 -sigalg SHA256withECDSA -validity 365 -dname "CN=localhost,OU=Unknown,O=Unknown,L=Unknown,S=Unknown,C=Unknown" -storetype JKS -keystore ectest.jks -storepass 123456 -ext KeyUsage:c=digitalSignature,keyCertSign > > ECDH: > keytool -genkeypair -alias ec -keyalg EC -keysize 256 -sigalg SHA256withECDSA -validity 365 -dname "CN=localhost,OU=Unknown,O=Unknown,L=Unknown,S=Unknown,C=Unknown" -storetype JKS -keystore ectest.jks -storepass 123456 -ext KeyUsage:c=keyAgreement,keyCertSign > > I'm not sure if keyCertSign is required on self-signed certificates, added it just in case. > > Tested on OpenJDK 11.0.6. > > Regards, > Daniel Jeli?ski > > > [1] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-52r2.pdf > [2] http://mail.openjdk.java.net/pipermail/security-dev/2017-May/015902.html > [3] https://gist.github.com/djelinski/b4543a3eb7ea66306044c08b41bba00f -------------- next part -------------- An HTML attachment was scrubbed... URL: From djelinski1 at gmail.com Sat Oct 31 17:41:48 2020 From: djelinski1 at gmail.com (=?UTF-8?Q?Daniel_Jeli=C5=84ski?=) Date: Sat, 31 Oct 2020 18:41:48 +0100 Subject: ECC Key Usage ignored In-Reply-To: References: Message-ID: Sure Xuelei. Filed 9067508 for the client issue, and 9067509 for the server one. Thanks! Daniel sob., 31 pa? 2020 o 17:23 Xue-Lei Fan napisa?(a): > Hi Daniel, > > Would you mind file a bug for the tracking? > > Xuelei > > On Oct 31, 2020, at 5:45 AM, Daniel Jeli?ski wrote: > > Verified that this behavior is still observed with JDK 16 EA 22. > > Client side code responsible for server certificate validation is located > in EndEntityChecker.checkTLSServer [1]. That code is not executed when the > certificate is trusted [2]. I believe this is a bug - I wouldn't choose to > accept a server certificate when that certificate was only meant for > signing other certificates, for example. > > Server side code responsible for cipher selection may need some > refactoring; currently the server selects the first cipher that passes > validation checks and for which keyManager.chooseServerAlias(keyType,...) > returns a key. The key manager does not have information about the intended > key usage (keyType is "EC" for both ECDH and ECDSA, and we don't know which > cipher is being checked), so it returns the same certificate when querying > for ECDH and ECDSA. > > Fortunately ECDH is not popular, and most clients won't even try to > negotiate it. Still, I believe this is a bug that should be addressed. > > As far as I could tell, there were no bugs reported for this in the bug > system. Should I report them there? > > If someone is interested in the server code, here's the interesting > portion of stack trace: > > chooseServerAlias:260, SunX509KeyManagerImpl (sun.security.ssl) > createServerPossession:288, X509Authentication$X509PossessionGenerator (sun.security.ssl) > createPossession:214, X509Authentication$X509PossessionGenerator (sun.security.ssl) > createPossession:90, X509Authentication (sun.security.ssl) > createPossessions:51, SSLKeyExchange (sun.security.ssl) > chooseCipherSuite:443, ServerHello$T12ServerHelloProducer (sun.security.ssl) > > > [1] > http://hg.openjdk.java.net/jdk/jdk/file/ee1d592a9f53/src/java.base/share/classes/sun/security/validator/EndEntityChecker.java#l276 > [2] > http://hg.openjdk.java.net/jdk/jdk/file/ee1d592a9f53/src/java.base/share/classes/sun/security/validator/Validator.java#l267 > > wt., 27 pa? 2020 o 18:44 Daniel Jeli?ski > napisa?(a): > >> Hi all, >> >> TL;DR: both SSL server and client ignore KeyUsage certificate extension >> when determining the list of available cipher suites. They shouldn't; >> KeyUsage is the only differentiator between ECDH and ECDSA certificates. >> >> Long version: >> I'm experimenting with ECC certificates on my Jetty server; when I >> created an ECC certificate and tested the server with nmap, I found that >> both ECDSA and ECDH cipher suites are enabled. I don't want ECDH ciphers, >> but I don't want to add explicit excludes either. >> >> Reading into NIST recommendations [1] I found that ECDSA certificates >> should define KeyUsage extension with value digitalSignature, vs ECDH which >> should use keyAgreement value. >> I experimented with both combinations of KeyValue, both resulted in the >> same set of ciphers being offered by the server. The client doesn't seem to >> care about KeyUsage either - it accepts connections even when the selected >> cipher doesn't match KeyUsage. >> >> Chrome browser doesn't support ECDH ciphers, but it does support ECDSA. >> When connecting to a Java server using ECDH certificate, it displays the >> error "ERR_SSL_KEY_USAGE_INCOMPATIBLE"; the server offers an ECDSA >> cipher suite, which is rejected by the browser. >> >> The issue was already reported by Bernd Eckenfels here [2], but as far as >> I can tell, it is not addressed yet; I was able to reproduce it using >> slightly modified code of this gist [3]. Certificates were generated using >> keytool commands: >> >> ECDSA: >> keytool -genkeypair -alias ec -keyalg EC -keysize 256 -sigalg >> SHA256withECDSA -validity 365 -dname >> "CN=localhost,OU=Unknown,O=Unknown,L=Unknown,S=Unknown,C=Unknown" >> -storetype JKS -keystore ectest.jks -storepass 123456 -ext >> KeyUsage:c=digitalSignature,keyCertSign >> >> ECDH: >> keytool -genkeypair -alias ec -keyalg EC -keysize 256 -sigalg >> SHA256withECDSA -validity 365 -dname >> "CN=localhost,OU=Unknown,O=Unknown,L=Unknown,S=Unknown,C=Unknown" >> -storetype JKS -keystore ectest.jks -storepass 123456 -ext >> KeyUsage:c=keyAgreement,keyCertSign >> >> I'm not sure if keyCertSign is required on self-signed certificates, >> added it just in case. >> >> Tested on OpenJDK 11.0.6. >> >> Regards, >> Daniel Jeli?ski >> >> >> [1] >> https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-52r2.pdf >> [2] >> http://mail.openjdk.java.net/pipermail/security-dev/2017-May/015902.html >> [3] https://gist.github.com/djelinski/b4543a3eb7ea66306044c08b41bba00f >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: