From Alan.Bateman at oracle.com Thu Jul 1 08:43:46 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 1 Jul 2021 09:43:46 +0100 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v2] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: <083bb049-3eb5-fb83-4846-29627c7faf05@oracle.com> On 30/06/2021 17:15, Jaikiran Pai wrote: > > I understand that Alan's suggestion holds good and we should have some > logic in place which switches to using a temp file once we notice that > the sizes we are dealing with can exceed some threshold, but I guess > that is something we need to do separately outside of this PR? My comment was mostly just to point out that it's only a partial fix and it will eventually fail with an OOME once the deflated size is too big for the BAOS. I don't have a strong opinion on whether the complete fix is done in one, two or many PRs but I think the first step could be to use the "useTempFile" path when the entry size is larger than some (10s of MB?) threshold. -Alan From jai.forums2013 at gmail.com Thu Jul 1 09:13:35 2021 From: jai.forums2013 at gmail.com (Jaikiran Pai) Date: Thu, 1 Jul 2021 14:43:35 +0530 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v2] In-Reply-To: <083bb049-3eb5-fb83-4846-29627c7faf05@oracle.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> <083bb049-3eb5-fb83-4846-29627c7faf05@oracle.com> Message-ID: <2c8f9b6e-80d8-95ae-62b0-5ca4d9d3cd22@gmail.com> Thank you Alan and Lance. I'll update this PR shortly with the proposed approach. -Jaikiran On 01/07/21 2:13 pm, Alan Bateman wrote: > On 30/06/2021 17:15, Jaikiran Pai wrote: >> >> I understand that Alan's suggestion holds good and we should have >> some logic in place which switches to using a temp file once we >> notice that the sizes we are dealing with can exceed some threshold, >> but I guess that is something we need to do separately outside of >> this PR? > > My comment was mostly just to point out that it's only a partial fix > and it will eventually fail with an OOME once the deflated size is too > big for the BAOS. I don't have a strong opinion on whether the > complete fix is done in one, two or many PRs but I think the first > step could be to use the "useTempFile" path when the entry size is > larger than some (10s of MB?) threshold. > > -Alan From alanb at openjdk.java.net Thu Jul 1 13:07:59 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Thu, 1 Jul 2021 13:07:59 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside In-Reply-To: References: Message-ID: On Sun, 27 Jun 2021 13:13:42 GMT, Jaikiran Pai wrote: > Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? > > As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). > > Alan notes in that issue that: > >> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. > > This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: > >> The elements returned by the iterator are in no specific order. Some file > systems maintain special links to the directory itself and the directory's > parent directory. Entries representing these links are not returned by the > iterator. > > > The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. > > A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. src/jdk.zipfs/share/classes/jdk/nio/zipfs/ZipFileSystem.java line 1971: > 1969: } > 1970: return false; > 1971: } It might be a bit clearer if ZipPath define isSelfOrParent(), that would avoid needing to expose the internal representation. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Thu Jul 1 13:25:32 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 1 Jul 2021 13:25:32 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v2] In-Reply-To: References: Message-ID: > Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? > > As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). > > Alan notes in that issue that: > >> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. > > This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: > >> The elements returned by the iterator are in no specific order. Some file > systems maintain special links to the directory itself and the directory's > parent directory. Entries representing these links are not returned by the > iterator. > > > The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. > > A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: implement review suggestion - move isSelfOrParent to ZipPath class ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4604/files - new: https://git.openjdk.java.net/jdk/pull/4604/files/344da6cb..3971ad6f Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4604&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4604&range=00-01 Stats: 20 lines in 2 files changed: 8 ins; 9 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/4604.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4604/head:pull/4604 PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Thu Jul 1 13:25:37 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 1 Jul 2021 13:25:37 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v2] In-Reply-To: References: Message-ID: On Thu, 1 Jul 2021 13:05:26 GMT, Alan Bateman wrote: >> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: >> >> implement review suggestion - move isSelfOrParent to ZipPath class > > src/jdk.zipfs/share/classes/jdk/nio/zipfs/ZipFileSystem.java line 1971: > >> 1969: } >> 1970: return false; >> 1971: } > > It might be a bit clearer if ZipPath define isSelfOrParent(), that would avoid needing to expose the internal representation. Done. I've updated the PR to implement this suggestion. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From github.com+1701815+mkarg at openjdk.java.net Thu Jul 1 21:59:33 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 1 Jul 2021 21:59:33 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v2] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Draft: Renaming i and separating code into several methods Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/ed49098a..7d18ca62 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=00-01 Stats: 108 lines in 1 file changed: 57 ins; 39 del; 12 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Fri Jul 2 06:20:29 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Fri, 2 Jul 2021 06:20:29 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v3] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: Draft: Renaming i and separating code into several methods Signed-off-by: Markus Karg ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/7d18ca62..47ee00a2 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=01-02 Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Fri Jul 2 06:20:30 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Fri, 2 Jul 2021 06:20:30 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v2] In-Reply-To: References: Message-ID: On Thu, 1 Jul 2021 21:59:33 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has updated the pull request incrementally with one additional commit since the last revision: > > Draft: Renaming i and separating code into several methods > > Signed-off-by: Markus Karg I have renamed `i` (and other ambiguous variables) and separated the implementation of `transferTo` into multiple methods. I will look into the test examples next to learn what I can reuse when writing tests for all those implementations. Thank you for all the kind input and the good ideas. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From alanb at openjdk.java.net Fri Jul 2 09:17:03 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 2 Jul 2021 09:17:03 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v2] In-Reply-To: References: Message-ID: On Thu, 1 Jul 2021 13:25:32 GMT, Jaikiran Pai wrote: >> Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? >> >> As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). >> >> Alan notes in that issue that: >> >>> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. >> >> This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: >> >>> The elements returned by the iterator are in no specific order. Some file >> systems maintain special links to the directory itself and the directory's >> parent directory. Entries representing these links are not returned by the >> iterator. >> >> >> The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. >> >> A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > implement review suggestion - move isSelfOrParent to ZipPath class The implementation change look good. I skimmed the test. A helper method that maps a Set to a modifiable Set would avoid the 10 usages of toString() in the setup, just a suggestion. Also I was puzzled by fs.getRootDirectories().iterator().next() and why this isn't fs.getPath("/"). ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Fri Jul 2 10:25:29 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 2 Jul 2021 10:25:29 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v3] In-Reply-To: References: Message-ID: > Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? > > As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). > > Alan notes in that issue that: > >> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. > > This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: > >> The elements returned by the iterator are in no specific order. Some file > systems maintain special links to the directory itself and the directory's > parent directory. Entries representing these links are not returned by the > iterator. > > > The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. > > A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: implement review suggestions: - reduce the toString() calls by creating a helper - fs.getPath("/") instead of fs.getRootDirectories().iterator().next() ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4604/files - new: https://git.openjdk.java.net/jdk/pull/4604/files/3971ad6f..8e77d289 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4604&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4604&range=01-02 Stats: 18 lines in 1 file changed: 6 ins; 0 del; 12 mod Patch: https://git.openjdk.java.net/jdk/pull/4604.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4604/head:pull/4604 PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Fri Jul 2 10:31:00 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 2 Jul 2021 10:31:00 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v3] In-Reply-To: References: Message-ID: On Fri, 2 Jul 2021 10:25:29 GMT, Jaikiran Pai wrote: >> Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? >> >> As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). >> >> Alan notes in that issue that: >> >>> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. >> >> This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: >> >>> The elements returned by the iterator are in no specific order. Some file >> systems maintain special links to the directory itself and the directory's >> parent directory. Entries representing these links are not returned by the >> iterator. >> >> >> The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. >> >> A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > implement review suggestions: > - reduce the toString() calls by creating a helper > - fs.getPath("/") instead of fs.getRootDirectories().iterator().next() >I skimmed the test. A helper method that maps a Set to a modifiable Set would avoid the 10 usages of toString() in the setup, just a suggestion. Done. I introduced a helper in the updated version of the PR. > Also I was puzzled by fs.getRootDirectories().iterator().next() and why this isn't fs.getPath("/"). When I started with a reproducer and then this test, I used the code that the reporter had attached in the JBS issue. That one had used "fs.getRootDirectories().iterator().next()" to show this issue. I just happened to carry that over into the test. I've updated the PR to now use fs.getPath("/") and also verified that this version of the test continues to fail without the fix proposed in this PR and passes with the changes. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Fri Jul 2 11:06:40 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 2 Jul 2021 11:06:40 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: > Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? > > As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). > > Alan notes in that issue that: > >> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. > > This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: > >> The elements returned by the iterator are in no specific order. Some file > systems maintain special links to the directory itself and the directory's > parent directory. Entries representing these links are not returned by the > iterator. > > > The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. > > A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: - Merge latest from upstream master branch to bring in fixes that might help fix the unrelated tier1 failures in Github Actions job runs - implement review suggestions: - reduce the toString() calls by creating a helper - fs.getPath("/") instead of fs.getRootDirectories().iterator().next() - implement review suggestion - move isSelfOrParent to ZipPath class - 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4604/files - new: https://git.openjdk.java.net/jdk/pull/4604/files/8e77d289..abae52bb Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4604&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4604&range=02-03 Stats: 10322 lines in 336 files changed: 6220 ins; 2802 del; 1300 mod Patch: https://git.openjdk.java.net/jdk/pull/4604.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4604/head:pull/4604 PR: https://git.openjdk.java.net/jdk/pull/4604 From lancea at openjdk.java.net Fri Jul 2 11:12:04 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Fri, 2 Jul 2021 11:12:04 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: On Fri, 2 Jul 2021 11:06:40 GMT, Jaikiran Pai wrote: >> Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? >> >> As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). >> >> Alan notes in that issue that: >> >>> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. >> >> This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: >> >>> The elements returned by the iterator are in no specific order. Some file >> systems maintain special links to the directory itself and the directory's >> parent directory. Entries representing these links are not returned by the >> iterator. >> >> >> The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. >> >> A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. > > Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: > > - Merge latest from upstream master branch to bring in fixes that might help fix the unrelated tier1 failures in Github Actions job runs > - implement review suggestions: > - reduce the toString() calls by creating a helper > - fs.getPath("/") instead of fs.getRootDirectories().iterator().next() > - implement review suggestion - move isSelfOrParent to ZipPath class > - 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside Hi Jaikiran, Consider: try (var os = Files.newOutputStream(ZIPFILE); ZipOutputStream zos = new ZipOutputStream(os)) { zos.putNextEntry(new ZipEntry("../Hello.txt")); zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); } With your proposed fix, you will only return "/" when you walk the the Zip file and we should also return "/Hello.txt" If. you resolve the path when the Inode entry is created: ` name(new ZipPath(null, normalize(name), true).getResolvedPath());` That should address the issue and also allow you to access the entry. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From jai.forums2013 at gmail.com Fri Jul 2 12:08:43 2021 From: jai.forums2013 at gmail.com (Jaikiran Pai) Date: Fri, 2 Jul 2021 17:38:43 +0530 Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: <3d068121-9800-972c-dbd6-cd46b25f6dde@gmail.com> Hello Lance, On 02/07/21 4:42 pm, Lance Andersen wrote: > Hi Jaikiran, > > Consider: > > > try (var os = Files.newOutputStream(ZIPFILE); > ZipOutputStream zos = new ZipOutputStream(os)) { > zos.putNextEntry(new ZipEntry("../Hello.txt")); > zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); > } > > > With your proposed fix, you will only return "/" when you walk the the Zip file and we should also return "/Hello.txt" Thank you for noticing this issue in my change and bringing this up. I have a question around this use case. Please consider a small variation to your example as below: try (var os = Files.newOutputStream(ZIPFILE); ???????????? ZipOutputStream zos = new ZipOutputStream(os)) { ??????????? zos.putNextEntry(new ZipEntry("../Hello.txt")); ??????????? zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); ??????????? zos.closeEntry(); ??????????? zos.putNextEntry(new ZipEntry("/Hello.txt")); ??? ??? ??? zos.write("Bye bye".getBytes(StandardCharsets.UTF_8)); ??? ??? ??? zos.closeEntry(); ??????? } Notice that I now have a zip/jar which has 2 differently named entries "../Hello.txt" and "/Hello.txt". This creates the archive without any issues and those 2 entries are noted in its listing. Now assuming someone walks over this jar using the ZipFileSystem, starting at root ("/"), what should be the expected output? The path(s) returned will end up being "/" and "/Hello.txt" but which resource is expected to be served in this case? The one which has "Hello World" in it or the one which has "Bye bye"? This example can be extended a bit more by introducing a "./Hello.txt", in the jar, with yet another different content in that entry. Is there some specification for this that I can check and adapt the test case accordingly? -Jaikiran From Alan.Bateman at oracle.com Fri Jul 2 13:37:15 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 2 Jul 2021 14:37:15 +0100 Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: <3d068121-9800-972c-dbd6-cd46b25f6dde@gmail.com> References: <3d068121-9800-972c-dbd6-cd46b25f6dde@gmail.com> Message-ID: On 02/07/2021 13:08, Jaikiran Pai wrote: > > Thank you for noticing this issue in my change and bringing this up. I > have a question around this use case. Please consider a small > variation to your example as below: > > try (var os = Files.newOutputStream(ZIPFILE); > ???????????? ZipOutputStream zos = new ZipOutputStream(os)) { > ??????????? zos.putNextEntry(new ZipEntry("../Hello.txt")); > ??????????? zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); > ??????????? zos.closeEntry(); > ??????????? zos.putNextEntry(new ZipEntry("/Hello.txt")); > ??? ??? ??? zos.write("Bye bye".getBytes(StandardCharsets.UTF_8)); > ??? ??? ??? zos.closeEntry(); > > ??????? } > > Notice that I now have a zip/jar which has 2 differently named entries > "../Hello.txt" and "/Hello.txt". This creates the archive without any > issues and those 2 entries are noted in its listing. Now assuming > someone walks over this jar using the ZipFileSystem, starting at root > ("/"), what should be the expected output? The path(s) returned will > end up being "/" and "/Hello.txt" but which resource is expected to be > served in this case? The one which has "Hello World" in it or the one > which has "Bye bye"? This example can be extended a bit more by > introducing a "./Hello.txt", in the jar, with yet another different > content in that entry. Is there some specification for this that I can > check and adapt the test case accordingly? This is an area where treating a zip file as a virtual file system doesn't quite work. We may have to look at zipfs ignoring entries that contain "." or ".." name elements or have it throw an exception if they are encountered. I think the main thing is that the APIs and access behaves consistently. -Alan From jpai at openjdk.java.net Fri Jul 2 14:08:17 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 2 Jul 2021 14:08:17 GMT Subject: RFR: 8233020: (fs) UnixFileSystemProvider should use StaticProperty.userDir(). Message-ID: <7jQFZOFRheEkl5XeEof4GWJ1tKvhcFIIPYQ_S7T5vJk=.f7b2a312-b999-4324-9f75-983cce410641@github.com> Can I please get a review for this change which addresses https://bugs.openjdk.java.net/browse/JDK-8233020? The commit in this PR updates `sun.nio.fs.UnixFileSystemProvider` to use the `jdk.internal.util.StaticProperty`, which was introduced in https://bugs.openjdk.java.net/browse/JDK-8066709, to get the value for the "user.dir" system property. Given the nature of this change, no new jtreg test has been added. Alan notes in JDK-8233020: > The changes in JDK-8066709 were okay to leave out the Unix file system provider because it was initialized very early in the startup at the time. So, with input from Roger, I generated the classloading logs of a trivial helloworld application, before and after this change. I used `-verbose:class` to generate them. public class Test { public static void main(final String[] args) throws Exception { System.out.println("Hello world"); } } The classloading logs showed that the order, both before and after the change, has the `jdk.internal.util.StaticProperty` loaded way before the `sun.nio.fs.UnixFileSystemProvider`. Given the way `jdk.internal.util.StaticProperty` class gets loaded in phase 1 of java.lang.System initialization[1], this classloading order also implies that `jdk.internal.util.StaticProperty` is initialized before any instance of `sun.nio.fs.UnixFileSystemProvider` gets created. I also checked a few versions of JDKs - 11, 15, 16 and even 8 (which seems to have this backported) and none of them show the `sun.nio.fs.UnixFileSystemProvider` loading before the `StaticProperty`. I wasn't able to build the version of a JDK (due to build tool incompatibilities with my local system) before the change in JDK-8066709 was introduced. So I don't have any version of JDK to compare where `UnixFileSystemProvider` was perhaps being loaded way too early to cause any issue. So in short, on the ordering front, IMO this cha nge shouldn't cause any issues that I can think of. For the sake of reference, a snippet of the classloading logs before and after this change is provided at the end of this comment. One other aspect to consider is the usage of "StaticProperty" itself. The javadoc of this class states: > {@link SecurityManager#checkPropertyAccess} is NOT checked > in these access methods. The caller of these methods should take care to ensure > that the returned property is not made accessible to untrusted code. > In this specific case, the `sun.nio.fs.UnixFileSystemProvider` is a public class. It now uses this `StaticProperty` to get the "user.dir" system property value and passes it on to the `newFileSystem(String dir)` method of its sub-classes. However, the `newFileSystem` method that's being called has package-private access, so no application specific sub-classes or any code outside of the JDK can override it and get access to the passed "user.dir" system property value. So the absence of security manager check for the property access in this case won't pose an issue, IMO. Following are the classloading logs: Before: line 252: [0.031s][info][class,load] jdk.internal.util.StaticProperty source: shared objects file [0.031s][info][class,load] java.io.FileInputStream source: shared objects file [0.031s][info][class,load] java.io.FileDescriptor source: shared objects file [0.031s][info][class,load] jdk.internal.access.JavaIOFileDescriptorAccess source: shared objects file .... [0.074s][info][class,load] java.lang.module.ModuleReader source: shared objects file [0.074s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemModuleReader source: shared objects file [0.074s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base [0.074s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: shared objects file [0.074s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: shared objects file [0.074s][info][class,load] java.nio.file.Paths source: shared objects file [0.074s][info][class,load] java.nio.file.FileSystems source: shared objects file [0.074s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder source: shared objects file [0.074s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder$1 source: shared objects file [0.074s][info][class,load] sun.nio.fs.DefaultFileSystemProvider source: shared objects file [0.074s][info][class,load] java.nio.file.spi.FileSystemProvider source: shared objects file [0.074s][info][class,load] sun.nio.fs.AbstractFileSystemProvider source: shared objects file line 579: [0.074s][info][class,load] sun.nio.fs.UnixFileSystemProvider source: shared objects file After: line 252: [0.037s][info][class,load] jdk.internal.util.StaticProperty source: shared objects file [0.037s][info][class,load] java.io.FileInputStream source: shared objects file [0.037s][info][class,load] java.io.FileDescriptor source: shared objects file ... [0.081s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base [0.081s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: shared objects file [0.081s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: shared objects file [0.081s][info][class,load] java.nio.file.Paths source: shared objects file [0.081s][info][class,load] java.nio.file.FileSystems source: shared objects file [0.081s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder source: shared objects file [0.081s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder$1 source: shared objects file [0.082s][info][class,load] sun.nio.fs.DefaultFileSystemProvider source: shared objects file [0.082s][info][class,load] java.nio.file.spi.FileSystemProvider source: shared objects file [0.082s][info][class,load] sun.nio.fs.AbstractFileSystemProvider source: shared objects file line 579: [0.082s][info][class,load] sun.nio.fs.UnixFileSystemProvider source: shared objects file [1] https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/System.java#L2089 ------------- Commit messages: - 8233020: (fs) UnixFileSystemProvider should use StaticProperty.userDir() Changes: https://git.openjdk.java.net/jdk/pull/4668/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4668&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8233020 Stats: 4 lines in 1 file changed: 1 ins; 2 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/4668.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4668/head:pull/4668 PR: https://git.openjdk.java.net/jdk/pull/4668 From lance.andersen at oracle.com Fri Jul 2 16:13:36 2021 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 2 Jul 2021 16:13:36 +0000 Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: <3d068121-9800-972c-dbd6-cd46b25f6dde@gmail.com> References: <3d068121-9800-972c-dbd6-cd46b25f6dde@gmail.com> Message-ID: On Jul 2, 2021, at 8:08 AM, Jaikiran Pai > wrote: Hello Lance, On 02/07/21 4:42 pm, Lance Andersen wrote: Hi Jaikiran, Consider: try (var os = Files.newOutputStream(ZIPFILE); ZipOutputStream zos = new ZipOutputStream(os)) { zos.putNextEntry(new ZipEntry("../Hello.txt")); zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); } With your proposed fix, you will only return "/" when you walk the the Zip file and we should also return "/Hello.txt" Thank you for noticing this issue in my change and bringing this up. I have a question around this use case. Please consider a small variation to your example as below: try (var os = Files.newOutputStream(ZIPFILE); ZipOutputStream zos = new ZipOutputStream(os)) { zos.putNextEntry(new ZipEntry("../Hello.txt")); zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); zos.closeEntry(); zos.putNextEntry(new ZipEntry("/Hello.txt")); zos.write("Bye bye".getBytes(StandardCharsets.UTF_8)); zos.closeEntry(); } Notice that I now have a zip/jar which has 2 differently named entries "../Hello.txt" and "/Hello.txt". This creates the archive without any issues and those 2 entries are noted in its listing. Now assuming someone walks over this jar using the ZipFileSystem, starting at root ("/"), what should be the expected output? The path(s) returned will end up being "/" and "/Hello.txt" but which resource is expected to be served in this case? The one which has "Hello World" in it or the one which has "Bye bye"? This example can be extended a bit more by introducing a "./Hello.txt", in the jar, with yet another different content in that entry. Is there some specification for this that I can check and adapt the test case accordingly? Consider Hello.zip created via: try (var os = Files.newOutputStream(Path.of("Hello.zip")); ZipOutputStream zos = new ZipOutputStream(os)) { zos.putNextEntry(new ZipEntry("../Hello.txt")); zos.write("Hello".getBytes(StandardCharsets.UTF_8)); zos.putNextEntry(new ZipEntry("Hello.txt")); zos.write("Another Hello".getBytes(StandardCharsets.UTF_8)); } Winzip does not handle this case well. InfoZip will : ------------- unzip ../Hello.zip Archive: ../Hello.zip warning: skipped "../" path component(s) in ../Hello.txt inflating: Hello.txt replace Hello.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: r new name: foo.txt inflating: foo.txt % ls Hello.txt foo.txt % cat Hello.txt Hello % cat foo.txt Another Hello % ???????? This scenario is not well covered in the Zip spec, so we can do what we think is best. Personally, I am OK with resolving the path. The odds of having multiple relative paths to the same file would be a mistake in creating the Zip. That being said, if we want to follow Alan?s suggestion and throw an Exception, I am OK with that as well. Either way, we currently cannot access the file via Zip FS due to the call to ZipPath::getResolvedPath() for all access and the path is only normalized when the Inodes are created. Alan, do you have a specific preference? -Jaikiran [cid:E1C4E2F0-ECD0-4C9D-ADB4-B16CA7BCB7FC at home] Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: oracle_sig_logo.gif URL: From lance.andersen at oracle.com Fri Jul 2 17:16:39 2021 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 2 Jul 2021 17:16:39 +0000 Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: <3d068121-9800-972c-dbd6-cd46b25f6dde@gmail.com> Message-ID: On Jul 2, 2021, at 12:13 PM, Lance Andersen > wrote: On Jul 2, 2021, at 8:08 AM, Jaikiran Pai > wrote: Hello Lance, On 02/07/21 4:42 pm, Lance Andersen wrote: Hi Jaikiran, Consider: try (var os = Files.newOutputStream(ZIPFILE); ZipOutputStream zos = new ZipOutputStream(os)) { zos.putNextEntry(new ZipEntry("../Hello.txt")); zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); } With your proposed fix, you will only return "/" when you walk the the Zip file and we should also return "/Hello.txt" Thank you for noticing this issue in my change and bringing this up. I have a question around this use case. Please consider a small variation to your example as below: try (var os = Files.newOutputStream(ZIPFILE); ZipOutputStream zos = new ZipOutputStream(os)) { zos.putNextEntry(new ZipEntry("../Hello.txt")); zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); zos.closeEntry(); zos.putNextEntry(new ZipEntry("/Hello.txt")); zos.write("Bye bye".getBytes(StandardCharsets.UTF_8)); zos.closeEntry(); } Notice that I now have a zip/jar which has 2 differently named entries "../Hello.txt" and "/Hello.txt". This creates the archive without any issues and those 2 entries are noted in its listing. Now assuming someone walks over this jar using the ZipFileSystem, starting at root ("/"), what should be the expected output? The path(s) returned will end up being "/" and "/Hello.txt" but which resource is expected to be served in this case? The one which has "Hello World" in it or the one which has "Bye bye"? This example can be extended a bit more by introducing a "./Hello.txt", in the jar, with yet another different content in that entry. Is there some specification for this that I can check and adapt the test case accordingly? Consider Hello.zip created via: try (var os = Files.newOutputStream(Path.of("Hello.zip")); ZipOutputStream zos = new ZipOutputStream(os)) { zos.putNextEntry(new ZipEntry("../Hello.txt")); zos.write("Hello".getBytes(StandardCharsets.UTF_8)); zos.putNextEntry(new ZipEntry("Hello.txt")); zos.write("Another Hello".getBytes(StandardCharsets.UTF_8)); } Winzip does not handle this case well. InfoZip will : ------------- unzip ../Hello.zip Archive: ../Hello.zip warning: skipped "../" path component(s) in ../Hello.txt inflating: Hello.txt replace Hello.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: r new name: foo.txt inflating: foo.txt % ls Hello.txt foo.txt % cat Hello.txt Hello % cat foo.txt Another Hello % ???????? This scenario is not well covered in the Zip spec, so we can do what we think is best. Personally, I am OK with resolving the path. The odds of having multiple relative paths to the same file would be a mistake in creating the Zip. That being said, if we want to follow Alan?s suggestion and throw an Exception, I am OK with that as well. Either way, we currently cannot access the file via Zip FS due to the call to ZipPath::getResolvedPath() for all access and the path is only normalized when the Inodes are created. Alan, do you have a specific preference? One more datapoint Consider: try (FileSystem zipfs = FileSystems.newFileSystem(Path.of("ZipFSHello.zip"), Map.of("create", "true"))) { Files.writeString(zipfs.getPath("HelloZipfs.txt"), "Hello"); Files.writeString(zipfs.getPath("../HelloZipfs.txt"), "A ZipFS Hello"); } The above will result in a Zip with a single entry of ?HelloZipfs.txt? and a value of ?A ZipFS Hello? As mentioned in my previous comment, this is due to the use of ZipPath::getResolvedPath() which brings me back to the suggestion of doing the same when we create the Inodes given this is an a virtual FS. HTH Best Lance -Jaikiran Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com [cid:E1C4E2F0-ECD0-4C9D-ADB4-B16CA7BCB7FC at home] Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: oracle_sig_logo.gif URL: From Alan.Bateman at oracle.com Fri Jul 2 18:29:19 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 2 Jul 2021 19:29:19 +0100 Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: <3d068121-9800-972c-dbd6-cd46b25f6dde@gmail.com> Message-ID: On 02/07/2021 18:16, Lance Andersen wrote: > : > > That being said, if we want to follow Alan?s suggestion and throw an Exception, I am OK with that as well. > > Either way, we currently cannot access the file via Zip FS due to the call to ZipPath::getResolvedPath() for all access and the path is only normalized when the Inodes are created. > > Alan, do you have a specific preference? > Not yet but if you tolerate these hostile entries then it means that all access with relative paths containing a "." or ".." element will be ambiguous. It will break relativize, normalize, and maybe other path operations. I assume it will break copy/move operations when the source is in a zip file and the target is in a non-zip file. I suspect it would also need to encode file names when open zip file for read/write access because the native file system is used for caching. -Alan From alanb at openjdk.java.net Sat Jul 3 07:28:52 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Sat, 3 Jul 2021 07:28:52 GMT Subject: RFR: 8233020: (fs) UnixFileSystemProvider should use StaticProperty.userDir(). In-Reply-To: <7jQFZOFRheEkl5XeEof4GWJ1tKvhcFIIPYQ_S7T5vJk=.f7b2a312-b999-4324-9f75-983cce410641@github.com> References: <7jQFZOFRheEkl5XeEof4GWJ1tKvhcFIIPYQ_S7T5vJk=.f7b2a312-b999-4324-9f75-983cce410641@github.com> Message-ID: On Fri, 2 Jul 2021 13:50:20 GMT, Jaikiran Pai wrote: > Can I please get a review for this change which addresses https://bugs.openjdk.java.net/browse/JDK-8233020? > > The commit in this PR updates `sun.nio.fs.UnixFileSystemProvider` to use the `jdk.internal.util.StaticProperty`, which was introduced in https://bugs.openjdk.java.net/browse/JDK-8066709, to get the value for the "user.dir" system property. Given the nature of this change, no new jtreg test has been added. > > Alan notes in JDK-8233020: > >> The changes in JDK-8066709 were okay to leave out the Unix file system provider because it was initialized very early in the startup at the time. > > So, with input from Roger, I generated the classloading logs of a trivial helloworld application, before and after this change. I used `-verbose:class` to generate them. > > > public class Test { > public static void main(final String[] args) throws Exception { > System.out.println("Hello world"); > } > } > > > The classloading logs showed that the order, both before and after the change, has the `jdk.internal.util.StaticProperty` loaded way before the `sun.nio.fs.UnixFileSystemProvider`. Given the way `jdk.internal.util.StaticProperty` class gets loaded in phase 1 of java.lang.System initialization[1], this classloading order also implies that `jdk.internal.util.StaticProperty` is initialized before any instance of `sun.nio.fs.UnixFileSystemProvider` gets created. I also checked a few versions of JDKs - 11, 15, 16 and even 8 (which seems to have this backported) and none of them show the `sun.nio.fs.UnixFileSystemProvider` loading before the `StaticProperty`. I wasn't able to build the version of a JDK (due to build tool incompatibilities with my local system) before the change in JDK-8066709 was introduced. So I don't have any version of JDK to compare where `UnixFileSystemProvider` was perhaps being loaded way too early to cause any issue. So in short, on the ordering front, IMO this c hange shouldn't cause any issues that I can think of. > > For the sake of reference, a snippet of the classloading logs before and after this change is provided at the end of this comment. > > One other aspect to consider is the usage of "StaticProperty" itself. The javadoc of this class states: > >> {@link SecurityManager#checkPropertyAccess} is NOT checked >> in these access methods. The caller of these methods should take care to ensure >> that the returned property is not made accessible to untrusted code. >> > > > In this specific case, the `sun.nio.fs.UnixFileSystemProvider` is a public class. It now uses this `StaticProperty` to get the "user.dir" system property value and passes it on to the `newFileSystem(String dir)` method of its sub-classes. However, the `newFileSystem` method that's being called has package-private access, so no application specific sub-classes or any code outside of the JDK can override it and get access to the passed "user.dir" system property value. So the absence of security manager check for the property access in this case won't pose an issue, IMO. > > Following are the classloading logs: > > Before: > > > line 252: [0.031s][info][class,load] jdk.internal.util.StaticProperty source: shared objects file > [0.031s][info][class,load] java.io.FileInputStream source: shared objects file > [0.031s][info][class,load] java.io.FileDescriptor source: shared objects file > [0.031s][info][class,load] jdk.internal.access.JavaIOFileDescriptorAccess source: shared objects file > .... > [0.074s][info][class,load] java.lang.module.ModuleReader source: shared objects file > [0.074s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemModuleReader source: shared objects file > [0.074s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base > [0.074s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: shared objects file > [0.074s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: shared objects file > [0.074s][info][class,load] java.nio.file.Paths source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder$1 source: shared objects file > [0.074s][info][class,load] sun.nio.fs.DefaultFileSystemProvider source: shared objects file > [0.074s][info][class,load] java.nio.file.spi.FileSystemProvider source: shared objects file > [0.074s][info][class,load] sun.nio.fs.AbstractFileSystemProvider source: shared objects file > line 579: [0.074s][info][class,load] sun.nio.fs.UnixFileSystemProvider source: shared objects file > > > After: > > > line 252: [0.037s][info][class,load] jdk.internal.util.StaticProperty source: shared objects file > [0.037s][info][class,load] java.io.FileInputStream source: shared objects file > [0.037s][info][class,load] java.io.FileDescriptor source: shared objects file > ... > [0.081s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base > [0.081s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: shared objects file > [0.081s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: shared objects file > [0.081s][info][class,load] java.nio.file.Paths source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder$1 source: shared objects file > [0.082s][info][class,load] sun.nio.fs.DefaultFileSystemProvider source: shared objects file > [0.082s][info][class,load] java.nio.file.spi.FileSystemProvider source: shared objects file > [0.082s][info][class,load] sun.nio.fs.AbstractFileSystemProvider source: shared objects file > line 579: [0.082s][info][class,load] sun.nio.fs.UnixFileSystemProvider source: shared objects file > > > > [1] https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/System.java#L2089 Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/4668 From jpai at openjdk.java.net Sat Jul 3 16:19:23 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Sat, 3 Jul 2021 16:19:23 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v4] In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: Introduce a way to allow outputstream returned by ZipOutputStream to rollover into a temporary file during writes, if the data size exceeds a threshold ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4607/files - new: https://git.openjdk.java.net/jdk/pull/4607/files/f519aa47..5c5f2694 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=02-03 Stats: 362 lines in 4 files changed: 330 ins; 18 del; 14 mod Patch: https://git.openjdk.java.net/jdk/pull/4607.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4607/head:pull/4607 PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Sat Jul 3 16:19:25 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Sat, 3 Jul 2021 16:19:25 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v3] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On Wed, 30 Jun 2021 16:05:40 GMT, Jaikiran Pai wrote: >> Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? >> >> The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. >> >> P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 > > Jaikiran Pai has updated the pull request incrementally with two additional commits since the last revision: > > - an initial proposed test for testing compressed size entry greater than 2gb > - minor summary update on test Based on the inputs received, I've now updated this PR to enhance the ZipFS to allow for rolling over the outputstream data, into a temporary file after reaching a threshold. Details follow: - A new "tempFileThreshold" property has been introduced for ZipFileSystem. This property can be passed like other existing properties when creating the filesystem. The value of this property is expected to be a size in bytes and represents the threshold which will be used to decide whether and when to use a temporary file for outputstreams of zip file entries returned by the ZipFileSystem. - The "tempFileThreshold" property is optional and if not set to any explicit value will default to 10MB. In other words, this feature is by default enabled, without the user having to do any specific configuration (not even the existing "useTempFile" property is requried to be set). - To disable the threshold based temp file creation feature, a value of 0 or a negative value can be passed. - A new (internal) `FileRolloverOutputStream` has been introduced in `ZipFileSystem` and forms the core of this enhancement. It extends the `ByteArrayOutputStream` and its write methods have the additional ability to rollover the current and subsequently written data into a temporary file representing that zip file entry. - A new `ZipFSOutputStreamTest` has been added with the sole focus of verifying the usage of this new "tempFileThreshold" property. - The previously added `LargeEntrySizeTest` and the manual `LargeCompressedEntrySizeTest` continue to stay and are mainly there to test the large file sizes and deflated sizes of the zip entries and verify that the original reported JBS issue is fixed by this enhancement. One important thing to note about these tests is that I've now removed the explicit "@requires" (memory) requirements, since after this enhancement (with "tempFileThreshold" by default enabled as in those tests), it no longer should require any specific high memory systems to run these tests. There are still some decisions to be made: 1. Should we introduce this new property or should we enhance the existing "useTempFile" property to allow a size to be passed? Specifically, can we/should we use that existing property to allow users to set the following values: - "true", this would imply the temp file feature is enabled always irrespective of the zip entry size. This is how the value of "true" is currently handled before the patch in this PR. So no change in behaviour. - a byte size, represented as a `String` or an integer. This would imply that the user wants to enable the temp file feature, but only when the size or compressed size of the zip entry reaches a threshold specified by this value. This would mean that for sizes lesser than this the ZipFS implementation would use a ByteArrayOutputStream and would only rollover to a temp file when the threshold is reached. - "false", this would disable the temp file feature completely and outputstreams for zip entries of the ZipFS instance will always use ByteArrayOutputStream - value of "0" or "-1". This would be the same as specifying "false" value. Using the existing property and just one property to control the temp file creation semantics will help avoid having to deal with 2 different properties ("useTempFile" and the "tempFileThreshold"). Plus it would also prevent any potentially conflicting user specified values for them. For example, we won't have to decide what to do when a user sets useTempFile=false and tempFileThreshold=1234. If we do decide to introduce this new property then some small amount of additional work needs to be done in this implementation to make sure semantically conflicting values for these 2 properties are handled correctly. I decided to skip that part in this round of the PR till we reached a decision about the properties. 2. Given that this is a new enhancement, I believe this requires a CSR. 3. Should this PR be now linked to https://bugs.openjdk.java.net/browse/JDK-8011146 instead of https://bugs.openjdk.java.net/browse/JDK-8190753? 4. I've never previously created a manual test case. The `LargeCompressedEntrySizeTest` in this PR is expected to be a manual test case (given how long it might take to run on various different systems). The only difference between this test case and other jtreg automated tests is the absence of a `@test` on this one. Is this how manual tests are written or is there some other way? ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Sun Jul 4 07:17:44 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Sun, 4 Jul 2021 07:17:44 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v5] In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: propagate back the original checked IOException to the callers ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4607/files - new: https://git.openjdk.java.net/jdk/pull/4607/files/5c5f2694..a9dfd8cd Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=03-04 Stats: 19 lines in 1 file changed: 12 ins; 0 del; 7 mod Patch: https://git.openjdk.java.net/jdk/pull/4607.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4607/head:pull/4607 PR: https://git.openjdk.java.net/jdk/pull/4607 From Alan.Bateman at oracle.com Sun Jul 4 15:17:26 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 4 Jul 2021 16:17:26 +0100 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v3] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On 03/07/2021 17:19, Jaikiran Pai wrote: > : > > > There are still some decisions to be made: > > 1. Should we introduce this new property or should we enhance the existing "useTempFile" property to allow a size to be passed? It's an implementation tuning knob and highly unlikely to be in the environment specified to newFileSystem. So I think the simplest is to just leave it out, it can always be added at a later time if there is a new real need. The default that you've chosen (10MB) looks a reasonable value to start with. BTW: The documented properties that can specified in the environment are in jdk.zipfs's module description. The useTempFile property is not listed here as it was never a documented/supported property that dates back to early iterations of the zip file system when it was a demo. As the PR has now expanded to do JDK-8011146 (and it doesn't matter if you use that JBS issue or continue with JDK-8190753) then I think the priority should be to be confident with the security. > 4. I've never previously created a manual test case. The `LargeCompressedEntrySizeTest` in this PR is expected to be a manual test case (given how long it might take to run on various different systems). The only difference between this test case and other jtreg automated tests is the absence of a `@test` on this one. Is this how manual tests are written or is there some other way? > We avoid manual tests as there is no guarantee that they will be run. So maybe we'll need to explore the scenario a bit further to see if there is some way to come up with an automated test. The jtreg foo for manual tests is `@run main/manual LargeCompressedEntrySizeTest`. You'll see a few examples in the test suite but I don't know if they are ever run. -Alan From jpai at openjdk.java.net Mon Jul 5 04:30:27 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Mon, 5 Jul 2021 04:30:27 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v6] In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 Jaikiran Pai has updated the pull request incrementally with three additional commits since the last revision: - remove no longer used constant - use jtreg's construct of manual test - Implement Alan's review suggestion - don't expose the threshold as a configuration property, instead set it internally to a specific value ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4607/files - new: https://git.openjdk.java.net/jdk/pull/4607/files/a9dfd8cd..50e64d74 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=04-05 Stats: 33 lines in 2 files changed: 1 ins; 25 del; 7 mod Patch: https://git.openjdk.java.net/jdk/pull/4607.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4607/head:pull/4607 PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Mon Jul 5 04:37:47 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Mon, 5 Jul 2021 04:37:47 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v7] In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: minor update to comment on FileRolloverOutputStream class ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4607/files - new: https://git.openjdk.java.net/jdk/pull/4607/files/50e64d74..0fdbcea6 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=05-06 Stats: 3 lines in 1 file changed: 0 ins; 3 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/4607.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4607/head:pull/4607 PR: https://git.openjdk.java.net/jdk/pull/4607 From jai.forums2013 at gmail.com Mon Jul 5 05:22:29 2021 From: jai.forums2013 at gmail.com (Jaikiran Pai) Date: Mon, 5 Jul 2021 10:52:29 +0530 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v3] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: Hello Alan, On 04/07/21 8:47 pm, Alan Bateman wrote: > On 03/07/2021 17:19, Jaikiran Pai wrote: >> : >> >> >> There are still some decisions to be made: >> >> 1. Should we introduce this new property or should we enhance the >> existing "useTempFile" property to allow a size to be passed? > It's an implementation tuning knob and highly unlikely to be in the > environment specified to newFileSystem. So I think the simplest is to > just leave it out, it can always be added at a later time if there is > a new real need. The default that you've chosen (10MB) looks a > reasonable value to start with. > > BTW: The documented properties that can specified in the environment > are in jdk.zipfs's module description. The useTempFile property is not > listed here as it was never a documented/supported property that dates > back to early iterations of the zip file system when it was a demo. I've updated the PR to make this an internal implementation detail/value and not expect it to be a configuration passed during the creation of the file system. This also allowed me to simplify some of this new code to no longer check for 0 or negative values for this property and other related semantics. > > As the PR has now expanded to do JDK-8011146 (and it doesn't matter if > you use that JBS issue or continue with JDK-8190753) then I think the > priority should be to be confident with the security. > Couple of things that I can think of when it comes to security is making sure the temp file is created in the right location and right set of permissions and making sure that the temp file isn't left behind after the filesystem is closed or after the zip entry is no longer relevant. When it comes to temp file creation, the code in this PR uses existing constructs/code that's present in this jdk.nio.zipfs.ZipFileSystem class. I will take a detailed look at this part once I get to my workstation later today. Are there any other security aspects that we need to think about? >> 4. I've never previously created a manual test case. The >> `LargeCompressedEntrySizeTest` in this PR is expected to be a manual >> test case (given how long it might take to run on various different >> systems). The only difference between this test case and other jtreg >> automated tests is the absence of a `@test` on this one. Is this how >> manual tests are written or is there some other way? >> > We avoid manual tests as there is no guarantee that they will be run. > So maybe we'll need to explore the scenario a bit further to see if > there is some way to come up with an automated test. The jtreg foo for > manual tests is `@run main/manual LargeCompressedEntrySizeTest`. > You'll see a few examples in the test suite but I don't know if they > are ever run. I have updated the PR to use jtreg's construct of @run testng/manual to mark this as a manual test. I will post the timing of this test case later today after I run the latest version locally and see how long it's taking. -Jaikiran From jpai at openjdk.java.net Mon Jul 5 07:42:26 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Mon, 5 Jul 2021 07:42:26 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v8] In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: reorganize the tests now that the temp file creation threshold isn't exposed as a user configurable value ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4607/files - new: https://git.openjdk.java.net/jdk/pull/4607/files/0fdbcea6..c1ec9e12 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=07 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=06-07 Stats: 200 lines in 3 files changed: 1 ins; 184 del; 15 mod Patch: https://git.openjdk.java.net/jdk/pull/4607.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4607/head:pull/4607 PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Mon Jul 5 13:57:53 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Mon, 5 Jul 2021 13:57:53 GMT Subject: RFR: 8233020: (fs) UnixFileSystemProvider should use StaticProperty.userDir(). In-Reply-To: <7jQFZOFRheEkl5XeEof4GWJ1tKvhcFIIPYQ_S7T5vJk=.f7b2a312-b999-4324-9f75-983cce410641@github.com> References: <7jQFZOFRheEkl5XeEof4GWJ1tKvhcFIIPYQ_S7T5vJk=.f7b2a312-b999-4324-9f75-983cce410641@github.com> Message-ID: <_uXYtR8NAD5Vv22-nCRhxG_9N0iY9BQBjFsmZQhMwf8=.f0e69c3c-f144-4f42-9fa1-c2b2a743cfe7@github.com> On Fri, 2 Jul 2021 13:50:20 GMT, Jaikiran Pai wrote: > Can I please get a review for this change which addresses https://bugs.openjdk.java.net/browse/JDK-8233020? > > The commit in this PR updates `sun.nio.fs.UnixFileSystemProvider` to use the `jdk.internal.util.StaticProperty`, which was introduced in https://bugs.openjdk.java.net/browse/JDK-8066709, to get the value for the "user.dir" system property. Given the nature of this change, no new jtreg test has been added. > > Alan notes in JDK-8233020: > >> The changes in JDK-8066709 were okay to leave out the Unix file system provider because it was initialized very early in the startup at the time. > > So, with input from Roger, I generated the classloading logs of a trivial helloworld application, before and after this change. I used `-verbose:class` to generate them. > > > public class Test { > public static void main(final String[] args) throws Exception { > System.out.println("Hello world"); > } > } > > > The classloading logs showed that the order, both before and after the change, has the `jdk.internal.util.StaticProperty` loaded way before the `sun.nio.fs.UnixFileSystemProvider`. Given the way `jdk.internal.util.StaticProperty` class gets loaded in phase 1 of java.lang.System initialization[1], this classloading order also implies that `jdk.internal.util.StaticProperty` is initialized before any instance of `sun.nio.fs.UnixFileSystemProvider` gets created. I also checked a few versions of JDKs - 11, 15, 16 and even 8 (which seems to have this backported) and none of them show the `sun.nio.fs.UnixFileSystemProvider` loading before the `StaticProperty`. I wasn't able to build the version of a JDK (due to build tool incompatibilities with my local system) before the change in JDK-8066709 was introduced. So I don't have any version of JDK to compare where `UnixFileSystemProvider` was perhaps being loaded way too early to cause any issue. So in short, on the ordering front, IMO this c hange shouldn't cause any issues that I can think of. > > For the sake of reference, a snippet of the classloading logs before and after this change is provided at the end of this comment. > > One other aspect to consider is the usage of "StaticProperty" itself. The javadoc of this class states: > >> {@link SecurityManager#checkPropertyAccess} is NOT checked >> in these access methods. The caller of these methods should take care to ensure >> that the returned property is not made accessible to untrusted code. >> > > > In this specific case, the `sun.nio.fs.UnixFileSystemProvider` is a public class. It now uses this `StaticProperty` to get the "user.dir" system property value and passes it on to the `newFileSystem(String dir)` method of its sub-classes. However, the `newFileSystem` method that's being called has package-private access, so no application specific sub-classes or any code outside of the JDK can override it and get access to the passed "user.dir" system property value. So the absence of security manager check for the property access in this case won't pose an issue, IMO. > > Following are the classloading logs: > > Before: > > > line 252: [0.031s][info][class,load] jdk.internal.util.StaticProperty source: shared objects file > [0.031s][info][class,load] java.io.FileInputStream source: shared objects file > [0.031s][info][class,load] java.io.FileDescriptor source: shared objects file > [0.031s][info][class,load] jdk.internal.access.JavaIOFileDescriptorAccess source: shared objects file > .... > [0.074s][info][class,load] java.lang.module.ModuleReader source: shared objects file > [0.074s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemModuleReader source: shared objects file > [0.074s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base > [0.074s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: shared objects file > [0.074s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: shared objects file > [0.074s][info][class,load] java.nio.file.Paths source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder$1 source: shared objects file > [0.074s][info][class,load] sun.nio.fs.DefaultFileSystemProvider source: shared objects file > [0.074s][info][class,load] java.nio.file.spi.FileSystemProvider source: shared objects file > [0.074s][info][class,load] sun.nio.fs.AbstractFileSystemProvider source: shared objects file > line 579: [0.074s][info][class,load] sun.nio.fs.UnixFileSystemProvider source: shared objects file > > > After: > > > line 252: [0.037s][info][class,load] jdk.internal.util.StaticProperty source: shared objects file > [0.037s][info][class,load] java.io.FileInputStream source: shared objects file > [0.037s][info][class,load] java.io.FileDescriptor source: shared objects file > ... > [0.081s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base > [0.081s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: shared objects file > [0.081s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: shared objects file > [0.081s][info][class,load] java.nio.file.Paths source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder$1 source: shared objects file > [0.082s][info][class,load] sun.nio.fs.DefaultFileSystemProvider source: shared objects file > [0.082s][info][class,load] java.nio.file.spi.FileSystemProvider source: shared objects file > [0.082s][info][class,load] sun.nio.fs.AbstractFileSystemProvider source: shared objects file > line 579: [0.082s][info][class,load] sun.nio.fs.UnixFileSystemProvider source: shared objects file > > > > [1] https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/System.java#L2089 Thank you for the review, Alan. ------------- PR: https://git.openjdk.java.net/jdk/pull/4668 From jpai at openjdk.java.net Mon Jul 5 13:57:53 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Mon, 5 Jul 2021 13:57:53 GMT Subject: Integrated: 8233020: (fs) UnixFileSystemProvider should use StaticProperty.userDir(). In-Reply-To: <7jQFZOFRheEkl5XeEof4GWJ1tKvhcFIIPYQ_S7T5vJk=.f7b2a312-b999-4324-9f75-983cce410641@github.com> References: <7jQFZOFRheEkl5XeEof4GWJ1tKvhcFIIPYQ_S7T5vJk=.f7b2a312-b999-4324-9f75-983cce410641@github.com> Message-ID: <8rpMKayl0nyLIH0p8QDLXLAWmMVCEdN6fwY4RTrWAj8=.7c7ee8e8-f8f1-459d-8b5c-24b46a213479@github.com> On Fri, 2 Jul 2021 13:50:20 GMT, Jaikiran Pai wrote: > Can I please get a review for this change which addresses https://bugs.openjdk.java.net/browse/JDK-8233020? > > The commit in this PR updates `sun.nio.fs.UnixFileSystemProvider` to use the `jdk.internal.util.StaticProperty`, which was introduced in https://bugs.openjdk.java.net/browse/JDK-8066709, to get the value for the "user.dir" system property. Given the nature of this change, no new jtreg test has been added. > > Alan notes in JDK-8233020: > >> The changes in JDK-8066709 were okay to leave out the Unix file system provider because it was initialized very early in the startup at the time. > > So, with input from Roger, I generated the classloading logs of a trivial helloworld application, before and after this change. I used `-verbose:class` to generate them. > > > public class Test { > public static void main(final String[] args) throws Exception { > System.out.println("Hello world"); > } > } > > > The classloading logs showed that the order, both before and after the change, has the `jdk.internal.util.StaticProperty` loaded way before the `sun.nio.fs.UnixFileSystemProvider`. Given the way `jdk.internal.util.StaticProperty` class gets loaded in phase 1 of java.lang.System initialization[1], this classloading order also implies that `jdk.internal.util.StaticProperty` is initialized before any instance of `sun.nio.fs.UnixFileSystemProvider` gets created. I also checked a few versions of JDKs - 11, 15, 16 and even 8 (which seems to have this backported) and none of them show the `sun.nio.fs.UnixFileSystemProvider` loading before the `StaticProperty`. I wasn't able to build the version of a JDK (due to build tool incompatibilities with my local system) before the change in JDK-8066709 was introduced. So I don't have any version of JDK to compare where `UnixFileSystemProvider` was perhaps being loaded way too early to cause any issue. So in short, on the ordering front, IMO this c hange shouldn't cause any issues that I can think of. > > For the sake of reference, a snippet of the classloading logs before and after this change is provided at the end of this comment. > > One other aspect to consider is the usage of "StaticProperty" itself. The javadoc of this class states: > >> {@link SecurityManager#checkPropertyAccess} is NOT checked >> in these access methods. The caller of these methods should take care to ensure >> that the returned property is not made accessible to untrusted code. >> > > > In this specific case, the `sun.nio.fs.UnixFileSystemProvider` is a public class. It now uses this `StaticProperty` to get the "user.dir" system property value and passes it on to the `newFileSystem(String dir)` method of its sub-classes. However, the `newFileSystem` method that's being called has package-private access, so no application specific sub-classes or any code outside of the JDK can override it and get access to the passed "user.dir" system property value. So the absence of security manager check for the property access in this case won't pose an issue, IMO. > > Following are the classloading logs: > > Before: > > > line 252: [0.031s][info][class,load] jdk.internal.util.StaticProperty source: shared objects file > [0.031s][info][class,load] java.io.FileInputStream source: shared objects file > [0.031s][info][class,load] java.io.FileDescriptor source: shared objects file > [0.031s][info][class,load] jdk.internal.access.JavaIOFileDescriptorAccess source: shared objects file > .... > [0.074s][info][class,load] java.lang.module.ModuleReader source: shared objects file > [0.074s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemModuleReader source: shared objects file > [0.074s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base > [0.074s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: shared objects file > [0.074s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: shared objects file > [0.074s][info][class,load] java.nio.file.Paths source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder source: shared objects file > [0.074s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder$1 source: shared objects file > [0.074s][info][class,load] sun.nio.fs.DefaultFileSystemProvider source: shared objects file > [0.074s][info][class,load] java.nio.file.spi.FileSystemProvider source: shared objects file > [0.074s][info][class,load] sun.nio.fs.AbstractFileSystemProvider source: shared objects file > line 579: [0.074s][info][class,load] sun.nio.fs.UnixFileSystemProvider source: shared objects file > > > After: > > > line 252: [0.037s][info][class,load] jdk.internal.util.StaticProperty source: shared objects file > [0.037s][info][class,load] java.io.FileInputStream source: shared objects file > [0.037s][info][class,load] java.io.FileDescriptor source: shared objects file > ... > [0.081s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base > [0.081s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: shared objects file > [0.081s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: shared objects file > [0.081s][info][class,load] java.nio.file.Paths source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder source: shared objects file > [0.081s][info][class,load] java.nio.file.FileSystems$DefaultFileSystemHolder$1 source: shared objects file > [0.082s][info][class,load] sun.nio.fs.DefaultFileSystemProvider source: shared objects file > [0.082s][info][class,load] java.nio.file.spi.FileSystemProvider source: shared objects file > [0.082s][info][class,load] sun.nio.fs.AbstractFileSystemProvider source: shared objects file > line 579: [0.082s][info][class,load] sun.nio.fs.UnixFileSystemProvider source: shared objects file > > > > [1] https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/lang/System.java#L2089 This pull request has now been integrated. Changeset: fd4de1ed Author: Jaikiran Pai URL: https://git.openjdk.java.net/jdk/commit/fd4de1ed404640ee0e744f022bbfa89db200ef05 Stats: 4 lines in 1 file changed: 1 ins; 2 del; 1 mod 8233020: (fs) UnixFileSystemProvider should use StaticProperty.userDir(). Reviewed-by: alanb ------------- PR: https://git.openjdk.java.net/jdk/pull/4668 From pconcannon at openjdk.java.net Tue Jul 6 10:24:29 2021 From: pconcannon at openjdk.java.net (Patrick Concannon) Date: Tue, 6 Jul 2021 10:24:29 GMT Subject: RFR: 8253119: Remove the legacy PlainSocketImpl and PlainDatagramSocketImpl implementation [v6] In-Reply-To: References: Message-ID: > Hi, > > Could someone please review my changes for the removal of the legacy `PlainSocketImpl` and `PlainDatagramSocketImpl` implementations? > > In JDK 13, JEP 353 provided a drop in replacement for the legacy `PlainSocketImpl` implementation. Since JDK 13, the `PlainSocketImpl` implementation was no longer used but included a mitigation mechanism to reduce compatibility risks in the form of a JDK-specific property `jdk.net.usePlainSocketImpl` allowing to switch back to the old implementation. > Similarly, in JDK 15, JEP 373 provided a new implementation for `DatagramSocket` and `MulticastSocket`, with a JDK-specific property `jdk.net.usePlainDatagramSocketImpl` also allowing the user to switch back to the old implementation in case of compatibility issue. > > As these implementations (and the mechanisms they use to enable them to mitigate compatibility issues) have been deemed no longer necessary, they now represent a maintenance burden. This patch looks at removing them from the JDK. > > Kind regards, > Patrick Patrick Concannon has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains seven additional commits since the last revision: - Merge remote-tracking branch 'origin/master' into JDK-8253119 - 8253119: Fixed typo - 8253119: Added message to null check in java/net/DatagramSocket - Merge remote-tracking branch 'origin/master' into JDK-8253119 - 8253119: Removed redundant comments and USE_PLAINDATAGRAMSOCKET from java/net/DatagramSocket - Merge remote-tracking branch 'origin/master' into JDK-8253119 - 8253119: Remove the legacy PlainSocketImpl and PlainDatagramSocketImpl implementation ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4574/files - new: https://git.openjdk.java.net/jdk/pull/4574/files/686b4369..538b145e Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4574&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4574&range=04-05 Stats: 12821 lines in 436 files changed: 7154 ins; 3504 del; 2163 mod Patch: https://git.openjdk.java.net/jdk/pull/4574.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4574/head:pull/4574 PR: https://git.openjdk.java.net/jdk/pull/4574 From pconcannon at openjdk.java.net Tue Jul 6 13:50:53 2021 From: pconcannon at openjdk.java.net (Patrick Concannon) Date: Tue, 6 Jul 2021 13:50:53 GMT Subject: Integrated: 8253119: Remove the legacy PlainSocketImpl and PlainDatagramSocketImpl implementation In-Reply-To: References: Message-ID: On Wed, 23 Jun 2021 12:06:41 GMT, Patrick Concannon wrote: > Hi, > > Could someone please review my changes for the removal of the legacy `PlainSocketImpl` and `PlainDatagramSocketImpl` implementations? > > In JDK 13, JEP 353 provided a drop in replacement for the legacy `PlainSocketImpl` implementation. Since JDK 13, the `PlainSocketImpl` implementation was no longer used but included a mitigation mechanism to reduce compatibility risks in the form of a JDK-specific property `jdk.net.usePlainSocketImpl` allowing to switch back to the old implementation. > Similarly, in JDK 15, JEP 373 provided a new implementation for `DatagramSocket` and `MulticastSocket`, with a JDK-specific property `jdk.net.usePlainDatagramSocketImpl` also allowing the user to switch back to the old implementation in case of compatibility issue. > > As these implementations (and the mechanisms they use to enable them to mitigate compatibility issues) have been deemed no longer necessary, they now represent a maintenance burden. This patch looks at removing them from the JDK. > > Kind regards, > Patrick This pull request has now been integrated. Changeset: 326b2e13 Author: Patrick Concannon URL: https://git.openjdk.java.net/jdk/commit/326b2e13447d734f84271942cc8154e30486fa7d Stats: 11059 lines in 78 files changed: 136 ins; 10853 del; 70 mod 8253119: Remove the legacy PlainSocketImpl and PlainDatagramSocketImpl implementation Reviewed-by: alanb, dfuchs, chegar ------------- PR: https://git.openjdk.java.net/jdk/pull/4574 From chegar at openjdk.java.net Tue Jul 6 14:51:06 2021 From: chegar at openjdk.java.net (Chris Hegarty) Date: Tue, 6 Jul 2021 14:51:06 GMT Subject: RFR: 8269917: Insert missing commas in copyrights in java.net In-Reply-To: References: Message-ID: On Tue, 6 Jul 2021 14:41:18 GMT, Patrick Concannon wrote: > Hi, > > Could someone please review my patch for Inserting missing commas in copyright headers from files in java.net. These commas were mistakenly left out of PR https://github.com/openjdk/jdk/pull/4574. > > Kind regards, > > Patrick Marked as reviewed by chegar (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/4690 From pconcannon at openjdk.java.net Tue Jul 6 14:51:05 2021 From: pconcannon at openjdk.java.net (Patrick Concannon) Date: Tue, 6 Jul 2021 14:51:05 GMT Subject: RFR: 8269917: Insert missing commas in copyrights in java.net Message-ID: Hi, Could someone please review my patch for Inserting missing commas in copyright headers from files in java.net. These commas were mistakenly left out of PR https://github.com/openjdk/jdk/pull/4574. Kind regards, Patrick ------------- Commit messages: - 8269917: Insert missing commas in copyrights in java.net Changes: https://git.openjdk.java.net/jdk/pull/4690/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4690&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8269917 Stats: 20 lines in 20 files changed: 0 ins; 0 del; 20 mod Patch: https://git.openjdk.java.net/jdk/pull/4690.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4690/head:pull/4690 PR: https://git.openjdk.java.net/jdk/pull/4690 From dfuchs at openjdk.java.net Tue Jul 6 14:51:06 2021 From: dfuchs at openjdk.java.net (Daniel Fuchs) Date: Tue, 6 Jul 2021 14:51:06 GMT Subject: RFR: 8269917: Insert missing commas in copyrights in java.net In-Reply-To: References: Message-ID: On Tue, 6 Jul 2021 14:41:18 GMT, Patrick Concannon wrote: > Hi, > > Could someone please review my patch for Inserting missing commas in copyright headers from files in java.net. These commas were mistakenly left out of PR https://github.com/openjdk/jdk/pull/4574. > > Kind regards, > > Patrick LGTM ------------- Marked as reviewed by dfuchs (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/4690 From pconcannon at openjdk.java.net Tue Jul 6 15:13:49 2021 From: pconcannon at openjdk.java.net (Patrick Concannon) Date: Tue, 6 Jul 2021 15:13:49 GMT Subject: Integrated: 8269917: Insert missing commas in copyrights in java.net In-Reply-To: References: Message-ID: On Tue, 6 Jul 2021 14:41:18 GMT, Patrick Concannon wrote: > Hi, > > Could someone please review my patch for Inserting missing commas in copyright headers from files in java.net. These commas were mistakenly left out of PR https://github.com/openjdk/jdk/pull/4574. > > Kind regards, > > Patrick This pull request has now been integrated. Changeset: 1578979d Author: Patrick Concannon URL: https://git.openjdk.java.net/jdk/commit/1578979df706ae9f4324931e1fc0d58265762c79 Stats: 20 lines in 20 files changed: 0 ins; 0 del; 20 mod 8269917: Insert missing commas in copyrights in java.net Reviewed-by: chegar, dfuchs ------------- PR: https://git.openjdk.java.net/jdk/pull/4690 From github.com+10835776+stsypanov at openjdk.java.net Thu Jul 8 09:32:02 2021 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Thu, 8 Jul 2021 09:32:02 GMT Subject: RFR: 8263561: Re-examine uses of LinkedList [v4] In-Reply-To: References: Message-ID: > After I've renamed remove branch GitHub for some reason has closed original https://github.com/openjdk/jdk/pull/2744, so I've decided to recreate it. ?????? ??????? has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains ten commits: - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 # Conflicts: # src/java.base/unix/classes/sun/net/dns/ResolverConfigurationImpl.java - Merge branch 'master' into purge-linked-list - 8263561: Use sized constructor where reasonable - 8263561: Use interface List instead of particular type where possible - 8263561: Rename requestList -> requests - 8263561: Re-examine uses of LinkedList ------------- Changes: https://git.openjdk.java.net/jdk/pull/4304/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4304&range=03 Stats: 48 lines in 9 files changed: 0 ins; 2 del; 46 mod Patch: https://git.openjdk.java.net/jdk/pull/4304.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4304/head:pull/4304 PR: https://git.openjdk.java.net/jdk/pull/4304 From sebastian.stenzel at gmail.com Thu Jul 8 14:54:29 2021 From: sebastian.stenzel at gmail.com (Sebastian Stenzel) Date: Thu, 8 Jul 2021 16:54:29 +0200 Subject: Make java.nio.ByteBuffer a sealed class Message-ID: A while back, when working on JDK-8264110, we had a discussion about the nature of ByteBuffers, where I asked whether we could eventually make this a sealed class. [1] The idea is, that basically there are only two kinds of ByteBuffers: - HeapByteBuffers - MappedByteBuffer ? DirectByteBuffer While the former is backed by an array, the latter isn't, which leads to branching such as `if (buffer.hasArray())`, which could be simplified, if we knew for sure to deal with either of the two kinds. Now, that sealed classes are stable with JDK 17, I'd like to ask for your approval to make ByteBuffer a sealed class, so we can rely on an exhaustive list of ByteBuffer "flavours" at compile time. This is a requirement for follow-up tasks, I'd like to work on, such as JDK-8264341. [1] https://github.com/openjdk/jdk/pull/3217#discussion_r602528233 From mullan at openjdk.java.net Thu Jul 8 18:28:12 2021 From: mullan at openjdk.java.net (Sean Mullan) Date: Thu, 8 Jul 2021 18:28:12 GMT Subject: [jdk17] RFR: 8266345: (fs) Custom DefaultFileSystemProvider security related loops Message-ID: Please review this fix to use the platform's default file system to avoid recursive policy initialization issues when a SecurityManager is enabled and the VM is configured to use a custom file system provider. ------------- Commit messages: - 8266345: (fs) Custom DefaultFileSystemProvider security related loops Changes: https://git.openjdk.java.net/jdk17/pull/232/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk17&pr=232&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8266345 Stats: 30 lines in 3 files changed: 28 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk17/pull/232.diff Fetch: git fetch https://git.openjdk.java.net/jdk17 pull/232/head:pull/232 PR: https://git.openjdk.java.net/jdk17/pull/232 From bpb at openjdk.java.net Thu Jul 8 18:39:54 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 8 Jul 2021 18:39:54 GMT Subject: [jdk17] RFR: 8266345: (fs) Custom DefaultFileSystemProvider security related loops In-Reply-To: References: Message-ID: On Thu, 8 Jul 2021 18:21:31 GMT, Sean Mullan wrote: > Please review this fix to use the platform's default file system to avoid recursive policy initialization > issues when a SecurityManager is enabled and the VM is configured to use a custom file system provider. Looks good. ------------- Marked as reviewed by bpb (Reviewer). PR: https://git.openjdk.java.net/jdk17/pull/232 From jai.forums2013 at gmail.com Fri Jul 9 12:43:44 2021 From: jai.forums2013 at gmail.com (Jaikiran Pai) Date: Fri, 9 Jul 2021 18:13:44 +0530 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v3] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On 05/07/21 10:52 am, Jaikiran Pai wrote: > >>> 4. I've never previously created a manual test case. The >>> `LargeCompressedEntrySizeTest` in this PR is expected to be a manual >>> test case (given how long it might take to run on various different >>> systems). The only difference between this test case and other jtreg >>> automated tests is the absence of a `@test` on this one. Is this how >>> manual tests are written or is there some other way? >>> >> We avoid manual tests as there is no guarantee that they will be run. >> So maybe we'll need to explore the scenario a bit further to see if >> there is some way to come up with an automated test. The jtreg foo >> for manual tests is `@run main/manual LargeCompressedEntrySizeTest`. >> You'll see a few examples in the test suite but I don't know if they >> are ever run. > > I have updated the PR to use jtreg's construct of @run testng/manual > to mark this as a manual test. I will post the timing of this test > case later today after I run the latest version locally and see how > long it's taking. > On my local setup, the LargeCompressedEntrySizeTest (latest version of this PR) consistently takes between 205 to 215 seconds to complete (so between 3 to 4 minutes). Is that something that will allow it to be added as part of automated tests or is it too long for automated tests? -Jaikiran From lance.andersen at oracle.com Fri Jul 9 16:11:48 2021 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 9 Jul 2021 16:11:48 +0000 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v3] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: <0FDCCD5F-4D35-46E8-B5E1-DEE83CC663A8@oracle.com> Hi Jaikiran, On Jul 9, 2021, at 8:43 AM, Jaikiran Pai > wrote: On 05/07/21 10:52 am, Jaikiran Pai wrote: 4. I've never previously created a manual test case. The `LargeCompressedEntrySizeTest` in this PR is expected to be a manual test case (given how long it might take to run on various different systems). The only difference between this test case and other jtreg automated tests is the absence of a `@test` on this one. Is this how manual tests are written or is there some other way? We avoid manual tests as there is no guarantee that they will be run. So maybe we'll need to explore the scenario a bit further to see if there is some way to come up with an automated test. The jtreg foo for manual tests is `@run main/manual LargeCompressedEntrySizeTest`. You'll see a few examples in the test suite but I don't know if they are ever run. I have updated the PR to use jtreg's construct of @run testng/manual to mark this as a manual test. I will post the timing of this test case later today after I run the latest version locally and see how long it's taking. On my local setup, the LargeCompressedEntrySizeTest (latest version of this PR) consistently takes between 205 to 215 seconds to complete (so between 3 to 4 minutes). Is that something that will allow it to be added as part of automated tests or is it too long for automated tests? Thank you for the info. This test needs to remain a manual test. I have a full test run going and will finish my review later today or tomorrow Best Lance -Jaikiran [cid:E1C4E2F0-ECD0-4C9D-ADB4-B16CA7BCB7FC at home] Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: oracle_sig_logo.gif URL: From lancea at openjdk.java.net Sun Jul 11 20:38:55 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Sun, 11 Jul 2021 20:38:55 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v8] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On Mon, 5 Jul 2021 07:42:26 GMT, Jaikiran Pai wrote: >> Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? >> >> The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. >> >> P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 > > Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: > > reorganize the tests now that the temp file creation threshold isn't exposed as a user configurable value I think the updates made to Zip FS look better. Alan is on vacation so I would prefer to wait until he gets back and give him a chance to provide any last thoughts on the change to Zip FS. The manual test looks OK and is a good addition ------------- Marked as reviewed by lancea (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/4607 From jai.forums2013 at gmail.com Mon Jul 12 01:19:47 2021 From: jai.forums2013 at gmail.com (Jaikiran Pai) Date: Mon, 12 Jul 2021 06:49:47 +0530 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v8] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: <9b331e83-3352-fbed-265b-38c6d2f22139@gmail.com> On 12/07/21 2:08 am, Lance Andersen wrote: > On Mon, 5 Jul 2021 07:42:26 GMT, Jaikiran Pai wrote: > >>> Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? >>> >>> The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. >>> >>> P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 >> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: >> >> reorganize the tests now that the temp file creation threshold isn't exposed as a user configurable value > I think the updates made to Zip FS look better. Alan is on vacation so I would prefer to wait until he gets back and give him a chance to provide any last thoughts on the change to Zip FS. > > The manual test looks OK and is a good addition Thank you for the reviews and running the tests, Lance. I'll wait for Alan to be back for his reviews. -Jaikiran From chegar at openjdk.java.net Mon Jul 12 14:42:52 2021 From: chegar at openjdk.java.net (Chris Hegarty) Date: Mon, 12 Jul 2021 14:42:52 GMT Subject: [jdk17] RFR: 8266345: (fs) Custom DefaultFileSystemProvider security related loops In-Reply-To: References: Message-ID: On Thu, 8 Jul 2021 18:21:31 GMT, Sean Mullan wrote: > Please review this fix to use the platform's default file system to avoid recursive policy initialization > issues when a SecurityManager is enabled and the VM is configured to use a custom file system provider. Marked as reviewed by chegar (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk17/pull/232 From mullan at openjdk.java.net Mon Jul 12 14:57:58 2021 From: mullan at openjdk.java.net (Sean Mullan) Date: Mon, 12 Jul 2021 14:57:58 GMT Subject: [jdk17] Integrated: 8266345: (fs) Custom DefaultFileSystemProvider security related loops In-Reply-To: References: Message-ID: On Thu, 8 Jul 2021 18:21:31 GMT, Sean Mullan wrote: > Please review this fix to use the platform's default file system to avoid recursive policy initialization > issues when a SecurityManager is enabled and the VM is configured to use a custom file system provider. This pull request has now been integrated. Changeset: 4fc3180f Author: Sean Mullan URL: https://git.openjdk.java.net/jdk17/commit/4fc3180f75e1cea4ebd613f8253be205d95f830c Stats: 30 lines in 3 files changed: 28 ins; 0 del; 2 mod 8266345: (fs) Custom DefaultFileSystemProvider security related loops Co-authored-by: Brian Burkhalter Reviewed-by: bpb, chegar ------------- PR: https://git.openjdk.java.net/jdk17/pull/232 From jwilhelm at openjdk.java.net Mon Jul 12 23:21:35 2021 From: jwilhelm at openjdk.java.net (Jesper Wilhelmsson) Date: Mon, 12 Jul 2021 23:21:35 GMT Subject: RFR: Merge jdk17 Message-ID: <7zQieFkCIJDip3Y4qUzNkzmvs5CWEaF2L09fXDMbLkc=.2b0d08b5-5ed6-4d63-bc76-fccad123bdb6@github.com> Forwardport JDK 17 -> JDK 18 ------------- Commit messages: - Merge - 8266345: (fs) Custom DefaultFileSystemProvider security related loops - 8269873: serviceability/sa/Clhsdb tests are using a C2 specific VMStruct field - 8268965: TCP Connection Reset when connecting simple socket to SSL server - 8269558: fix of JDK-8252657 missed to update history at the end of JVM TI spec - 8270216: [macOS] Update named used for Java run loop mode The webrevs contain the adjustments done while merging with regards to each parent branch: - master: https://webrevs.openjdk.java.net/?repo=jdk&pr=4760&range=00.0 - jdk17: https://webrevs.openjdk.java.net/?repo=jdk&pr=4760&range=00.1 Changes: https://git.openjdk.java.net/jdk/pull/4760/files Stats: 39 lines in 7 files changed: 33 ins; 2 del; 4 mod Patch: https://git.openjdk.java.net/jdk/pull/4760.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4760/head:pull/4760 PR: https://git.openjdk.java.net/jdk/pull/4760 From jwilhelm at openjdk.java.net Tue Jul 13 10:54:07 2021 From: jwilhelm at openjdk.java.net (Jesper Wilhelmsson) Date: Tue, 13 Jul 2021 10:54:07 GMT Subject: Integrated: Merge jdk17 In-Reply-To: <7zQieFkCIJDip3Y4qUzNkzmvs5CWEaF2L09fXDMbLkc=.2b0d08b5-5ed6-4d63-bc76-fccad123bdb6@github.com> References: <7zQieFkCIJDip3Y4qUzNkzmvs5CWEaF2L09fXDMbLkc=.2b0d08b5-5ed6-4d63-bc76-fccad123bdb6@github.com> Message-ID: On Mon, 12 Jul 2021 23:12:29 GMT, Jesper Wilhelmsson wrote: > Forwardport JDK 17 -> JDK 18 This pull request has now been integrated. Changeset: 6b123b05 Author: Jesper Wilhelmsson URL: https://git.openjdk.java.net/jdk/commit/6b123b059136b0c1efa62a23824b9aa253e6a519 Stats: 39 lines in 7 files changed: 33 ins; 2 del; 4 mod Merge ------------- PR: https://git.openjdk.java.net/jdk/pull/4760 From github.com+1701815+mkarg at openjdk.java.net Wed Jul 14 17:48:17 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Wed, 14 Jul 2021 17:48:17 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v3] In-Reply-To: References: Message-ID: <8kSiVkKwbP0EbOfpVHO9HLMPHM58ySG-ZLwZjMdCpUk=.ff8a4d7b-085d-487d-ba47-f072a439ce6a@github.com> On Fri, 2 Jul 2021 06:20:29 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. I cloned the `InputStream/TransferTo` test in a way which uses providers, but when started to implement the providers I noticed that a huge amount of *empty* source code is needed just to make the compiler happy: `FileChannel` already has dozens of abstract methods I have to override but they never will be used by any of the tests... Having that said, it would be good if I would be allowed to spare us thousands of useless codelines: * May I just implement the `content()` test or do you insist on me really implementing *all* the tests found in `InputStream/TransferTo` for *each* of the overloaded `transferTo` implementations? * May I use a mocking framework to actually just provide *partial* implementations of the channels, or do you insist on me actually overloading *all* methods of *all* channels in actual Java source code just for the sake of making the compiler happy? While I understand the necessity of tests, and while I am convinced that the performance benefit outweighs the weeks of work this would imply just for adding all those test code, I actually like to propose that I only cover the `content()` tests plus using Mockito, so we would have 80% of the benefit for 20% of the coding costs. WDYT? ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From bpb at openjdk.java.net Wed Jul 14 18:38:15 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Wed, 14 Jul 2021 18:38:15 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v3] In-Reply-To: References: Message-ID: On Fri, 2 Jul 2021 06:20:29 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: > > Draft: Renaming i and separating code into several methods > > Signed-off-by: Markus Karg Mockito is not approved for distribution and/or hosting by Java (Oracle JDK / OpenJDK). I understand not wanting to have a lot of useless code. Have you looked into other tests which have implemented Providers? ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 17 08:37:52 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 17 Jul 2021 08:37:52 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v3] In-Reply-To: References: Message-ID: On Wed, 14 Jul 2021 18:35:15 GMT, Brian Burkhalter wrote: > Have you looked into other tests which have implemented Providers? I have not found any test code containing the word `new FileChannel() {...}` or `extends FileChannel`, so I apparently there exists no mock of `FileChannel`. Unfortunately, as long as it is mandatory to copy *all* the tests of `InputStream/TransferTo` this means that I *must* implement *all* methods of `FileChannel` (even if they are empty) just for the sake of throwing an exception after one, two, or three transferred bytes (which, BTW, is not part of the API but looks like just a personal decision of the original test author). It is not really helpful that OpenJDK *neither* uses mocking *nor* lets me reduce the number of tests to the *essential* minimum, as this means, I wrote approx. 100 LoC for the optimized implementations and now I have to write approx. 1000 LoC just to perform *exactly the same tests* (to be clear: I have no problem with testing `content()` which certainly *has* to be tested, but which does *not* imply writing a `FileChannel` mock). ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Thu Jul 22 05:54:51 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 22 Jul 2021 05:54:51 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v3] In-Reply-To: References: Message-ID: <5JT_y4R4xZAA8CgrqeCsxrlbDkn2Q8817kyRymvn4F8=.e200a0b0-67e4-4742-97df-2c71000bc0c2@github.com> On Fri, 2 Jul 2021 06:20:29 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: > > Draft: Renaming i and separating code into several methods > > Signed-off-by: Markus Karg Understood. Work is in progress. Stay tuned. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From jpai at openjdk.java.net Thu Jul 22 12:55:46 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 22 Jul 2021 12:55:46 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v8] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On Wed, 21 Jul 2021 04:09:23 GMT, Jaikiran Pai wrote: > > > For some context - the new `FileRolloverOutputStream` extends `ByteArrayOutputStream` and hence cannot have a `throws IOException` in its overridden `write` methods. > > > > > > Have you tried wrapping a BAOS rather than extending, that might allow the exception wrapping/unwapping to go away. > > Hello Alan, > > I did experiment with it earlier, before going with the current approach in this PR. The disadvantage, as I see it, with wrapping a `ByteArrayOutputStream` instead of extending it is that when trying to rollover the contents to a file, you don't have access to the (inner protected) byte array of the ByteArrayOutputStream. > > The rollover code would then look something like: > > ``` > private void transferToFile() throws IOException { > // create a tempfile > entry.file = getTempPathForEntry(null); > // transfer the already written data from the byte array buffer into this tempfile > try (OutputStream os = new BufferedOutputStream(Files.newOutputStream(entry.file))) { > new ByteArrayInputStream(baos.toByteArray(), 0, baos.size()).transferTo(os); > } > // release the underlying byte array > baos = null; > // append any further data to the file with buffering enabled > tmpFileOS = new BufferedOutputStream(Files.newOutputStream(entry.file, APPEND)); > } > ``` > > So although you can transfer the contents to the file without requiring the access to the byte array, you end up creating a new copy of that array (through the use of `baos.toByteArray()`), which can be at most 10MB in size. I thought avoiding a new copy of that (potentially 10MB) array during this transfer would be a good save and hence decided to settle on extending `ByteArrayOutputStream` instead of wrapping it. > > The use of `extends` of course now means dealing with the `UncheckedIOException` as done in this PR. But if you think that the array copy isn't a concern and wrapping the `ByteArrayOutputStream` is a better way, then I'll go ahead and update this PR accordingly. Now that the mailing lists integration seems to be back to normal, just adding this dummy comment to bring to attention the latest comments in this PR. ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From ecki at zusammenkunft.net Thu Jul 22 15:24:39 2021 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Thu, 22 Jul 2021 15:24:39 +0000 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v8] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> , Message-ID: Hello, > So although you can transfer the contents to the file without requiring the access > to the byte array, you end up creating a new copy of that array (through the use > of `baos.toByteArray()`) You can avoid the copy and the additional buffer with baos.writeTo() I think. try (OutputStream os = Files.newOutputStream(entry.file)) { // maybe append? baos.writeTo(os); } // release the underlying byte array baos = null; // append any further data to the file with buffering enabled tmpFileOS = new BufferedOutputStream(Files.newOutputStream(entry.file, APPEND)); -- http://bernd.eckenfels.net ________________________________ Von: nio-dev im Auftrag von Jaikiran Pai Gesendet: Thursday, July 22, 2021 2:55:46 PM An: core-libs-dev at openjdk.java.net ; nio-dev at openjdk.java.net Betreff: Re: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v8] On Wed, 21 Jul 2021 04:09:23 GMT, Jaikiran Pai wrote: > > > For some context - the new `FileRolloverOutputStream` extends `ByteArrayOutputStream` and hence cannot have a `throws IOException` in its overridden `write` methods. > > > > > > Have you tried wrapping a BAOS rather than extending, that might allow the exception wrapping/unwapping to go away. > > Hello Alan, > > I did experiment with it earlier, before going with the current approach in this PR. The disadvantage, as I see it, with wrapping a `ByteArrayOutputStream` instead of extending it is that when trying to rollover the contents to a file, you don't have access to the (inner protected) byte array of the ByteArrayOutputStream. > > The rollover code would then look something like: > > ``` > private void transferToFile() throws IOException { > // create a tempfile > entry.file = getTempPathForEntry(null); > // transfer the already written data from the byte array buffer into this tempfile > try (OutputStream os = new BufferedOutputStream(Files.newOutputStream(entry.file))) { > new ByteArrayInputStream(baos.toByteArray(), 0, baos.size()).transferTo(os); > } > // release the underlying byte array > baos = null; > // append any further data to the file with buffering enabled > tmpFileOS = new BufferedOutputStream(Files.newOutputStream(entry.file, APPEND)); > } > ``` > > So although you can transfer the contents to the file without requiring the access to the byte array, you end up creating a new copy of that array (through the use of `baos.toByteArray()`), which can be at most 10MB in size. I thought avoiding a new copy of that (potentially 10MB) array during this transfer would be a good save and hence decided to settle on extending `ByteArrayOutputStream` instead of wrapping it. > > The use of `extends` of course now means dealing with the `UncheckedIOException` as done in this PR. But if you think that the array copy isn't a concern and wrapping the `ByteArrayOutputStream` is a better way, then I'll go ahead and update this PR accordingly. Now that the mailing lists integration seems to be back to normal, just adding this dummy comment to bring to attention the latest comments in this PR. ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jai.forums2013 at gmail.com Thu Jul 22 16:01:46 2021 From: jai.forums2013 at gmail.com (Jaikiran Pai) Date: Thu, 22 Jul 2021 21:31:46 +0530 Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v8] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: Hello Bernd, On 22/07/21 8:54 pm, Bernd Eckenfels wrote: > Hello, > >> So although you can transfer the contents to the file without requiring the access >> to the byte array, you end up creating a new copy of that array (through the use >> of `baos.toByteArray()`) > You can avoid the copy and the additional buffer with baos.writeTo() I think. > > try (OutputStream os = Files.newOutputStream(entry.file)) { // maybe append? > baos.writeTo(os); > } You are absolutely right. I hadn't noticed ByteArrayOutputStream had this writeTo() method. Thank you for this input. This was the only concern I had when it came to wrapping the ByteArrayOutputStream and now with your input it no longer is a concern. I have updated this PR to go ahead with the wrapping approach which also does away with the necessity of UncheckedIOException. Existing and the new tests continue to pass with this change. -Jaikiran From jpai at openjdk.java.net Thu Jul 22 16:01:30 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Thu, 22 Jul 2021 16:01:30 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v9] In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 Jaikiran Pai has updated the pull request incrementally with two additional commits since the last revision: - remove no longer necessary javadoc comment on test - review suggestion - wrap ByteArrayOutputStream instead of extending it ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4607/files - new: https://git.openjdk.java.net/jdk/pull/4607/files/c1ec9e12..90101d45 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=08 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=07-08 Stats: 72 lines in 2 files changed: 19 ins; 35 del; 18 mod Patch: https://git.openjdk.java.net/jdk/pull/4607.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4607/head:pull/4607 PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Fri Jul 23 03:54:13 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 03:54:13 GMT Subject: RFR: 8271147: java/nio/file/Path.java javadoc typo Message-ID: Can I please get a review for this change which fixes the typo noted in https://bugs.openjdk.java.net/browse/JDK-8271147? `make docs-image` worked fine and the generated javadoc looks fine. ------------- Commit messages: - 8271147: java/nio/file/Path.java javadoc typo Changes: https://git.openjdk.java.net/jdk/pull/4884/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4884&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8271147 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/4884.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4884/head:pull/4884 PR: https://git.openjdk.java.net/jdk/pull/4884 From iris at openjdk.java.net Fri Jul 23 04:01:06 2021 From: iris at openjdk.java.net (Iris Clark) Date: Fri, 23 Jul 2021 04:01:06 GMT Subject: RFR: 8271147: java/nio/file/Path.java javadoc typo In-Reply-To: References: Message-ID: On Fri, 23 Jul 2021 03:37:44 GMT, Jaikiran Pai wrote: > Can I please get a review for this change which fixes the typo noted in https://bugs.openjdk.java.net/browse/JDK-8271147? `make docs-image` worked fine and the generated javadoc looks fine. Marked as reviewed by iris (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/4884 From jpai at openjdk.java.net Fri Jul 23 04:10:09 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 04:10:09 GMT Subject: Integrated: 8271147: java/nio/file/Path.java javadoc typo In-Reply-To: References: Message-ID: On Fri, 23 Jul 2021 03:37:44 GMT, Jaikiran Pai wrote: > Can I please get a review for this change which fixes the typo noted in https://bugs.openjdk.java.net/browse/JDK-8271147? `make docs-image` worked fine and the generated javadoc looks fine. This pull request has now been integrated. Changeset: 8156ff60 Author: Jaikiran Pai URL: https://git.openjdk.java.net/jdk/commit/8156ff609b27316f31ba89d9eb8ca752f4027c2b Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod 8271147: java/nio/file/Path.java javadoc typo Reviewed-by: iris ------------- PR: https://git.openjdk.java.net/jdk/pull/4884 From jpai at openjdk.java.net Fri Jul 23 04:10:08 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 04:10:08 GMT Subject: RFR: 8271147: java/nio/file/Path.java javadoc typo In-Reply-To: References: Message-ID: On Fri, 23 Jul 2021 03:37:44 GMT, Jaikiran Pai wrote: > Can I please get a review for this change which fixes the typo noted in https://bugs.openjdk.java.net/browse/JDK-8271147? `make docs-image` worked fine and the generated javadoc looks fine. Thank you for the review, Iris. ------------- PR: https://git.openjdk.java.net/jdk/pull/4884 From alanb at openjdk.java.net Fri Jul 23 08:37:06 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 23 Jul 2021 08:37:06 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: On Fri, 2 Jul 2021 11:08:43 GMT, Lance Andersen wrote: >> Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: >> >> - Merge latest from upstream master branch to bring in fixes that might help fix the unrelated tier1 failures in Github Actions job runs >> - implement review suggestions: >> - reduce the toString() calls by creating a helper >> - fs.getPath("/") instead of fs.getRootDirectories().iterator().next() >> - implement review suggestion - move isSelfOrParent to ZipPath class >> - 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside > > Hi Jaikiran, > > Consider: > > > try (var os = Files.newOutputStream(ZIPFILE); > ZipOutputStream zos = new ZipOutputStream(os)) { > zos.putNextEntry(new ZipEntry("../Hello.txt")); > zos.write("Hello World".getBytes(StandardCharsets.UTF_8)); > } > > > With your proposed fix, you will only return "/" when you walk the the Zip file and we should also return "/Hello.txt" > > If. you resolve the path when the Inode entry is created: > > ` name(new ZipPath(null, normalize(name), true).getResolvedPath());` > > That should address the issue and also allow you to access the entry. I discussed this issue with @LanceAndersen and I think we are in agreement that a zip file system needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. The other options on the table (including Lance's origin prototype fix and the proposal in this PR) lead to anomalies. So I think we should close this PR and restart JDK-8251329. @jaikiran Are you okay this this? JDK-8251329 is still assigned to JDK-8251329 and has a patch with the changes that we think are the right way for newFileSystem to reject ZIP files that have entries in the CEN that can't be used for files in a file system. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Fri Jul 23 08:51:09 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 08:51:09 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: On Fri, 2 Jul 2021 11:06:40 GMT, Jaikiran Pai wrote: >> Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? >> >> As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). >> >> Alan notes in that issue that: >> >>> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. >> >> This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: >> >>> The elements returned by the iterator are in no specific order. Some file >> systems maintain special links to the directory itself and the directory's >> parent directory. Entries representing these links are not returned by the >> iterator. >> >> >> The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. >> >> A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. > > Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: > > - Merge latest from upstream master branch to bring in fixes that might help fix the unrelated tier1 failures in Github Actions job runs > - implement review suggestions: > - reduce the toString() calls by creating a helper > - fs.getPath("/") instead of fs.getRootDirectories().iterator().next() > - implement review suggestion - move isSelfOrParent to ZipPath class > - 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside Hello Alan, > So I think we should close this PR and restart JDK-8251329. @jaikiran Are you okay this this? Yes, this sounds fine to me. I don't have any objection in closing this PR and I'll do it shortly. > JDK-8251329 is still assigned to JDK-8251329 and has a patch with the changes that we think are the right way for newFileSystem to reject ZIP files This part I didn't understand. Did you mean to refer some other JBS issue? Because from what I see in https://bugs.openjdk.java.net/browse/JDK-8251329 there's no patch attached to it (unless of course it's restricted to specific JBS user roles?). ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Fri Jul 23 09:03:13 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 09:03:13 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: <7-Xlx5esvz244se9IG2KZDmLwscSh2SKk2dwSU1-l_0=.1a5b3fcb-a586-4c02-93fd-d45ef54fd78e@github.com> On Fri, 2 Jul 2021 11:06:40 GMT, Jaikiran Pai wrote: >> Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? >> >> As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). >> >> Alan notes in that issue that: >> >>> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. >> >> This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: >> >>> The elements returned by the iterator are in no specific order. Some file >> systems maintain special links to the directory itself and the directory's >> parent directory. Entries representing these links are not returned by the >> iterator. >> >> >> The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. >> >> A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. > > Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision: > > - Merge latest from upstream master branch to bring in fixes that might help fix the unrelated tier1 failures in Github Actions job runs > - implement review suggestions: > - reduce the toString() calls by creating a helper > - fs.getPath("/") instead of fs.getRootDirectories().iterator().next() > - implement review suggestion - move isSelfOrParent to ZipPath class > - 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside Closing this PR based on the above discussion. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Fri Jul 23 09:03:13 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 09:03:13 GMT Subject: Withdrawn: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside In-Reply-To: References: Message-ID: On Sun, 27 Jun 2021 13:13:42 GMT, Jaikiran Pai wrote: > Can I please get a review of this proposed fix for https://bugs.openjdk.java.net/browse/JDK-8251329? > > As noted in that issue, if a zip filesystem created on top of a jar containing a "./" entry is "walked" using the `Files.walkFileTree`, it leads to a infinite never ending iteration (which ultimately fails with Java heap space OOM). > > Alan notes in that issue that: > >> This is more likely an issue with the zipfs DirectoryStream implementation. A DirectoryStream is specified to not include elements that for the special links to the current or parent directory. It should be rare. > > This indeed turned out to be an issue in the `jdk.nio.zipfs.ZipDirectoryStream#iterator()` implementation where it calls the `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` implementation. The implementation, unlike what the javadoc of `java.nio.file.DirectoryStream` states, was including the "." and ".." paths in its returned iterator: > >> The elements returned by the iterator are in no specific order. Some file > systems maintain special links to the directory itself and the directory's > parent directory. Entries representing these links are not returned by the > iterator. > > > The proposed fix in this patch checks the paths for "." and "..", similar to what the `sun.nio.fs.UnixDirectoryStream` does in its `isSelfOrParent` and skips those paths from being added into the returned iterator. The `jdk.nio.zipfs.ZipFileSystem#iteratorOf(...)` (where this change has been done) is currently only used by `jdk.nio.zipfs.ZipDirectoryStream#iterator()` and has package-private visibility, so this change shouldn't impact any other usage/expectations. > > A new jtreg test has been added to reproduce this issue and verify the fix. Local testing of the `test/jdk/jdk/nio/` (including this new test) went fine without any issues after this change. I will be triggering a `tier1` test locally in a while. This pull request has been closed without being integrated. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From alanb at openjdk.java.net Fri Jul 23 10:31:04 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 23 Jul 2021 10:31:04 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v9] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On Thu, 22 Jul 2021 16:01:30 GMT, Jaikiran Pai wrote: >> Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? >> >> The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. >> >> P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 > > Jaikiran Pai has updated the pull request incrementally with two additional commits since the last revision: > > - remove no longer necessary javadoc comment on test > - review suggestion - wrap ByteArrayOutputStream instead of extending it Thanks for changing it to wrap the BOAS rather than existing it, that avoids the annoying wrapping/unwrapping of exceptions. So I think the approach looks good but I think the synchronization needs to be re-checked it is not obvious that is correct or needed. Are there any cases where FileRolloverOutputStream is returned to user code? I don't think so, instead users of the zip file system will get an EntryOutputStream that wraps the FileRolloverOutputStream. The EntryOutputStream methods are synchronized so I assume that FileRolloverOutputStream does not need to it and that would avoid the inconsistency between the write methods and the flush/close methods. One other thing to point out is that transferToFile shouldn't need to open the file twice, instead it should be able to open the tmp file for writing once. ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From alanb at openjdk.java.net Fri Jul 23 10:40:09 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 23 Jul 2021 10:40:09 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: On Fri, 23 Jul 2021 08:47:46 GMT, Jaikiran Pai wrote: > This part I didn't understand. Did you mean to refer some other JBS issue? Because from what I see in https://bugs.openjdk.java.net/browse/JDK-8251329 there's no patch attached to it (unless of course it's restricted to specific JBS user roles?). I wasn't suggesting there is a patch attached to that issue. Rather I was just pointing out that JDK-8251329 was being worked on already before this PR was created. Lance's initial patch for JDK-8251329 changed ZipPath::getResolvedPath when creating the inode entry but that approach leads to anomalies. Overall I think the discussion has been useful but I don't think it's feasible to have a file system view over entries that look like links to the current or parent directory. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From sebastian.stenzel at gmail.com Fri Jul 23 10:59:57 2021 From: sebastian.stenzel at gmail.com (Sebastian Stenzel) Date: Fri, 23 Jul 2021 12:59:57 +0200 Subject: Make java.nio.ByteBuffer a sealed class In-Reply-To: References: Message-ID: <1CC33B30-7CB8-4499-90A6-6749C37B3C6E@gmail.com> Any thoughts on this? > On 8. Jul 2021, at 16:54, Sebastian Stenzel wrote: > > A while back, when working on JDK-8264110, we had a discussion about the nature of ByteBuffers, where I asked whether we could eventually make this a sealed class. [1] > > The idea is, that basically there are only two kinds of ByteBuffers: > - HeapByteBuffers > - MappedByteBuffer ? DirectByteBuffer > > While the former is backed by an array, the latter isn't, which leads to branching such as `if (buffer.hasArray())`, which could be simplified, if we knew for sure to deal with either of the two kinds. > > Now, that sealed classes are stable with JDK 17, I'd like to ask for your approval to make ByteBuffer a sealed class, so we can rely on an exhaustive list of ByteBuffer "flavours" at compile time. > > This is a requirement for follow-up tasks, I'd like to work on, such as JDK-8264341. > > > [1] https://github.com/openjdk/jdk/pull/3217#discussion_r602528233 From jpai at openjdk.java.net Fri Jul 23 11:52:47 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 11:52:47 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v10] In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision: Implement review suggestions: - remove unnecessary "synchronized" - no need to open the temp file twice ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4607/files - new: https://git.openjdk.java.net/jdk/pull/4607/files/90101d45..9e78ba06 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=09 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=08-09 Stats: 12 lines in 1 file changed: 5 ins; 4 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/4607.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4607/head:pull/4607 PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Fri Jul 23 12:00:08 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 12:00:08 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v9] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On Fri, 23 Jul 2021 10:28:02 GMT, Alan Bateman wrote: > So I think the approach looks good but I think the synchronization needs to be re-checked it is not obvious that is correct or needed. Are there any cases where FileRolloverOutputStream is returned to user code? I don't think so, instead users of the zip file system will get an EntryOutputStream that wraps the FileRolloverOutputStream. The EntryOutputStream methods are synchronized so I assume that FileRolloverOutputStream does not need to it and that would avoid the inconsistency between the write methods and the flush/close methods. I hadn't paid any thoughts on the "synchronized" part. You are right - the new `FileRolloverOutputStream` doesn't get sent back to the callers directly and instead either the `EntryOutputStream` or the `DeflatingEntryOutputStream` get returned. Both of them have the necessary syncrhonizations in place for `write` and `close` operations. The `flush` of the `FileRolloverOutputStream` calls the `flush` on the `BufferedOutputStream` which already has the necessary synchronization. I've updated the PR to remove the use of `synchronized` from this new class and added a brief note about this for future maintainers, just like the existing `EntryOutputStreamDef` has. > One other thing to point out is that transferToFile shouldn't need to open the file twice, instead it should be able to open the tmp file for writing once. The updated version of this PR now fixes this part to open it just once. I had reviewed this `transferTo` multiple times before, but clearly I overlooked this part of the implementation. Thank you for these inputs. The updated PR continues to pass the new tests and the existing ones in `test/jdk/jdk/nio/zipfs/`. ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Fri Jul 23 12:09:10 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 12:09:10 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: On Fri, 23 Jul 2021 10:36:51 GMT, Alan Bateman wrote: > I wasn't suggesting there is a patch attached to that issue. Rather I was just pointing out that JDK-8251329 was being worked on already before this PR was created. Ok, that makes sense. Thank you for the details. ------------- PR: https://git.openjdk.java.net/jdk/pull/4604 From jpai at openjdk.java.net Fri Jul 23 14:58:01 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 14:58:01 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v11] In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 15 additional commits since the last revision: - Merge latest from master branch - Implement review suggestions: - remove unnecessary "synchronized" - no need to open the temp file twice - remove no longer necessary javadoc comment on test - review suggestion - wrap ByteArrayOutputStream instead of extending it - reorganize the tests now that the temp file creation threshold isn't exposed as a user configurable value - minor update to comment on FileRolloverOutputStream class - remove no longer used constant - use jtreg's construct of manual test - Implement Alan's review suggestion - don't expose the threshold as a configuration property, instead set it internally to a specific value - propagate back the original checked IOException to the callers - ... and 5 more: https://git.openjdk.java.net/jdk/compare/d783ad9d...991de6b9 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4607/files - new: https://git.openjdk.java.net/jdk/pull/4607/files/9e78ba06..991de6b9 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=10 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4607&range=09-10 Stats: 43946 lines in 1112 files changed: 20903 ins; 17871 del; 5172 mod Patch: https://git.openjdk.java.net/jdk/pull/4607.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4607/head:pull/4607 PR: https://git.openjdk.java.net/jdk/pull/4607 From alanb at openjdk.java.net Fri Jul 23 15:18:06 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 23 Jul 2021 15:18:06 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v9] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On Fri, 23 Jul 2021 11:57:34 GMT, Jaikiran Pai wrote: > The updated version of this PR now fixes this part to open it just once. I had reviewed this `transferTo` multiple times before, but clearly I overlooked this part of the implementation. > > Thank you for these inputs. The updated PR continues to pass the new tests and the existing ones in `test/jdk/jdk/nio/zipfs/`. The updated implementation looks okay, I don't think I have any more questions. ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Fri Jul 23 15:26:19 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 23 Jul 2021 15:26:19 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v11] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: <2ttMZAIqSLz7XfsDi0CQjsaApF19Vk68hNNgh_WDIw0=.a4b275b9-3ce9-4101-9dfd-0db920386eb1@github.com> On Fri, 23 Jul 2021 14:58:01 GMT, Jaikiran Pai wrote: >> Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? >> >> The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. >> >> P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 > > Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 15 additional commits since the last revision: > > - Merge latest from master branch > - Implement review suggestions: > - remove unnecessary "synchronized" > - no need to open the temp file twice > - remove no longer necessary javadoc comment on test > - review suggestion - wrap ByteArrayOutputStream instead of extending it > - reorganize the tests now that the temp file creation threshold isn't exposed as a user configurable value > - minor update to comment on FileRolloverOutputStream class > - remove no longer used constant > - use jtreg's construct of manual test > - Implement Alan's review suggestion - don't expose the threshold as a configuration property, instead set it internally to a specific value > - propagate back the original checked IOException to the callers > - ... and 5 more: https://git.openjdk.java.net/jdk/compare/e1196067...991de6b9 Thank you for the review Alan. @LanceAndersen, I've run the tier1 tests locally with the latest PR and they have passed without any regressions. Given that we changed the implementation to wrap ByteArrayOutputStream instead of extending it, would you want to rerun some of your other tests that you had previously run, before I integrate this? ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From paul.sandoz at oracle.com Fri Jul 23 15:46:25 2021 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Fri, 23 Jul 2021 15:46:25 +0000 Subject: Make java.nio.ByteBuffer a sealed class In-Reply-To: <1CC33B30-7CB8-4499-90A6-6749C37B3C6E@gmail.com> References: <1CC33B30-7CB8-4499-90A6-6749C37B3C6E@gmail.com> Message-ID: <11344009-DD3A-411F-93E5-0121DC95180D@oracle.com> I don?t see much advantage to this. Most of the (byte) buffer hierarchy is package private. Further, the hierarchy is set up for convenience: DirectByteBuffer extends MappedByteBuffer [*], even though not all DirectByteBuffer instances are mapped. Internally we can assume if a ByteBuffer instance is also an instance sun.nio.ch.DirectBuffer (which in turn implies that ByteBuffer.isDirect() == true) then it?s a direct buffer. Otherwise, its a heap buffer and if hasArray() == true then the underlying array is accessible. In the referenced code there is a precondition that the byte buffer is not read-only. Therefore, we can assume for a heap buffer the src array is accessible (use an assert if you wish). My hope is this kind of code can improve when we can use MemorySegment. Paul. [*] public abstract class MappedByteBuffer extends ByteBuffer { // This is a little bit backwards: By rights MappedByteBuffer should be a // subclass of DirectByteBuffer, but to keep the spec clear and simple, and // for optimization purposes, it's easier to do it the other way around. // This works because DirectByteBuffer is a package-private class. > On Jul 23, 2021, at 3:59 AM, Sebastian Stenzel wrote: > > Any thoughts on this? > >> On 8. Jul 2021, at 16:54, Sebastian Stenzel wrote: >> >> A while back, when working on JDK-8264110, we had a discussion about the nature of ByteBuffers, where I asked whether we could eventually make this a sealed class. [1] >> >> The idea is, that basically there are only two kinds of ByteBuffers: >> - HeapByteBuffers >> - MappedByteBuffer ? DirectByteBuffer >> >> While the former is backed by an array, the latter isn't, which leads to branching such as `if (buffer.hasArray())`, which could be simplified, if we knew for sure to deal with either of the two kinds. >> >> Now, that sealed classes are stable with JDK 17, I'd like to ask for your approval to make ByteBuffer a sealed class, so we can rely on an exhaustive list of ByteBuffer "flavours" at compile time. >> >> This is a requirement for follow-up tasks, I'd like to work on, such as JDK-8264341. >> >> >> [1] https://github.com/openjdk/jdk/pull/3217#discussion_r602528233 > From lancea at openjdk.java.net Fri Jul 23 16:25:03 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Fri, 23 Jul 2021 16:25:03 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v11] In-Reply-To: <2ttMZAIqSLz7XfsDi0CQjsaApF19Vk68hNNgh_WDIw0=.a4b275b9-3ce9-4101-9dfd-0db920386eb1@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> <2ttMZAIqSLz7XfsDi0CQjsaApF19Vk68hNNgh_WDIw0=.a4b275b9-3ce9-4101-9dfd-0db920386eb1@github.com> Message-ID: <27fDIfBum2vSv7DmFs_PH8Z27NAYMVeMUmAa6OOJgfI=.b125d8c7-83a0-48d5-8342-d0da696fdbff@github.com> On Fri, 23 Jul 2021 15:23:07 GMT, Jaikiran Pai wrote: > Thank you for the review Alan. > > @LanceAndersen, I've run the tier1 tests locally with the latest PR and they have passed without any regressions. Given that we changed the implementation to wrap ByteArrayOutputStream instead of extending it, would you want to rerun some of your other tests that you had previously run, before I integrate this? Yes, I will run additional tests and report back after they complete ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From sebastian.stenzel at gmail.com Fri Jul 23 21:48:31 2021 From: sebastian.stenzel at gmail.com (Sebastian Stenzel) Date: Fri, 23 Jul 2021 23:48:31 +0200 Subject: Make java.nio.ByteBuffer a sealed class In-Reply-To: <11344009-DD3A-411F-93E5-0121DC95180D@oracle.com> References: <11344009-DD3A-411F-93E5-0121DC95180D@oracle.com> Message-ID: Sure, we can just assert that any ByteBuffer that isn't direct must be a HeapByteBuffer. But this assertion is based on our knowlege. The point is, that we could use a mechanism which would instead cause a compile time error if this ever changes. In my opinion this is the stronger tool to ensure correctness than relying on every contributor to know that only these existing implementations of ByteBuffers are allowed. This change has no direct functional effect, it merely helps to reduce branches that are technically dead code and just exist to satisfy ByteBuffer's contract. In other words: There is no functional advantage. This is about reducing technical debt. > Am 23.07.2021 um 17:46 schrieb Paul Sandoz : > > ?I don?t see much advantage to this. Most of the (byte) buffer hierarchy is package private. Further, the hierarchy is set up for convenience: DirectByteBuffer extends MappedByteBuffer [*], even though not all DirectByteBuffer instances are mapped. > > Internally we can assume if a ByteBuffer instance is also an instance sun.nio.ch.DirectBuffer (which in turn implies that ByteBuffer.isDirect() == true) then it?s a direct buffer. Otherwise, its a heap buffer and if hasArray() == true then the underlying array is accessible. > > In the referenced code there is a precondition that the byte buffer is not read-only. Therefore, we can assume for a heap buffer the src array is accessible (use an assert if you wish). > > My hope is this kind of code can improve when we can use MemorySegment. > > Paul. > > [*] > public abstract class MappedByteBuffer > extends ByteBuffer > { > > // This is a little bit backwards: By rights MappedByteBuffer should be a > // subclass of DirectByteBuffer, but to keep the spec clear and simple, and > // for optimization purposes, it's easier to do it the other way around. > // This works because DirectByteBuffer is a package-private class. > >> On Jul 23, 2021, at 3:59 AM, Sebastian Stenzel wrote: >> >> Any thoughts on this? >> >>>> On 8. Jul 2021, at 16:54, Sebastian Stenzel wrote: >>> >>> A while back, when working on JDK-8264110, we had a discussion about the nature of ByteBuffers, where I asked whether we could eventually make this a sealed class. [1] >>> >>> The idea is, that basically there are only two kinds of ByteBuffers: >>> - HeapByteBuffers >>> - MappedByteBuffer ? DirectByteBuffer >>> >>> While the former is backed by an array, the latter isn't, which leads to branching such as `if (buffer.hasArray())`, which could be simplified, if we knew for sure to deal with either of the two kinds. >>> >>> Now, that sealed classes are stable with JDK 17, I'd like to ask for your approval to make ByteBuffer a sealed class, so we can rely on an exhaustive list of ByteBuffer "flavours" at compile time. >>> >>> This is a requirement for follow-up tasks, I'd like to work on, such as JDK-8264341. >>> >>> >>> [1] https://github.com/openjdk/jdk/pull/3217#discussion_r602528233 >> > From lancea at openjdk.java.net Sun Jul 25 22:04:40 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Sun, 25 Jul 2021 22:04:40 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside Message-ID: Hi, As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. Mach5 tiers 1 through 3 have been run without any errors encountered . Best, Lance ------------- Commit messages: - Address missing linefeed after package name - Address overzelous intellij import update - Patch to address JDK-8251329 Changes: https://git.openjdk.java.net/jdk/pull/4900/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8251329 Stats: 174 lines in 2 files changed: 174 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/4900.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4900/head:pull/4900 PR: https://git.openjdk.java.net/jdk/pull/4900 From alanb at openjdk.java.net Mon Jul 26 07:54:06 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Mon, 26 Jul 2021 07:54:06 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside In-Reply-To: References: Message-ID: On Sun, 25 Jul 2021 21:56:10 GMT, Lance Andersen wrote: > Hi, > > As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS > > Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. > > > Mach5 tiers 1 through 3 have been run without any errors encountered . > > Best, > Lance This is behavior change (to reject zip/JAR files) so a CSR will be required. I've also added a label to the JBS issue to remind us to add a release note. I think it would be good for @jaikiran to review too. src/jdk.zipfs/share/classes/jdk/nio/zipfs/ZipFileSystem.java line 1573: > 1571: } > 1572: IndexNode inode = new IndexNode(cen, pos, nlen); > 1573: if(hasDotOrDotDot(inode.name)) { Minor nit, missing space in "if(". src/jdk.zipfs/share/classes/jdk/nio/zipfs/ZipFileSystem.java line 1602: > 1600: // Inode.name always includes "/" in path[0] > 1601: if (path.length == 1) { > 1602: return false; It may be useful to add "assert path[0] == '/';" at the start of this method. test/jdk/jdk/nio/zipfs/HasDotDotTest.java line 1: > 1: import org.testng.annotations.DataProvider; Missing copyright header. ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From github.com+10835776+stsypanov at openjdk.java.net Mon Jul 26 08:27:20 2021 From: github.com+10835776+stsypanov at openjdk.java.net (=?UTF-8?B?0KHQtdGA0LPQtdC5?= =?UTF-8?B?IA==?= =?UTF-8?B?0KbRi9C/0LDQvdC+0LI=?=) Date: Mon, 26 Jul 2021 08:27:20 GMT Subject: RFR: 8263561: Re-examine uses of LinkedList [v5] In-Reply-To: References: Message-ID: > After I've renamed remove branch GitHub for some reason has closed original https://github.com/openjdk/jdk/pull/2744, so I've decided to recreate it. ?????? ??????? has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 10 commits: - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 - Merge branch 'master' into 8263561 # Conflicts: # src/java.base/unix/classes/sun/net/dns/ResolverConfigurationImpl.java - Merge branch 'master' into purge-linked-list - 8263561: Use sized constructor where reasonable - 8263561: Use interface List instead of particular type where possible - 8263561: Rename requestList -> requests - 8263561: Re-examine uses of LinkedList ------------- Changes: https://git.openjdk.java.net/jdk/pull/4304/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4304&range=04 Stats: 48 lines in 9 files changed: 0 ins; 2 del; 46 mod Patch: https://git.openjdk.java.net/jdk/pull/4304.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4304/head:pull/4304 PR: https://git.openjdk.java.net/jdk/pull/4304 From jpai at openjdk.java.net Mon Jul 26 09:55:07 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Mon, 26 Jul 2021 09:55:07 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside In-Reply-To: References: Message-ID: On Sun, 25 Jul 2021 21:56:10 GMT, Lance Andersen wrote: > Hi, > > As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS > > Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. > > > Mach5 tiers 1 through 3 have been run without any errors encountered . > > Best, > Lance This change looks fine to me. I was unsure how the writing/creating entries with `.` or `..` with `ZipFileSystem` would behave in context of this change, so I gave this a try locally with the changes from this PR: try (FileSystem zipfs = FileSystems.newFileSystem(ZIPFILE)) { final Path[] paths = new Path[]{ zipfs.getPath("./Hello.txt"), zipfs.getPath("../../../Hello.txt"), zipfs.getPath("../Hello.txt")}; for (int i = 0; i < paths.length; i++) { try (OutputStream os = Files.newOutputStream(paths[i])) { os.write(("Hello " + i).getBytes(StandardCharsets.UTF_8)); } } } This code runs fine and it ends up creating (just one) CEN entry for `Hello.txt`: JTwork/scratch/zipfsDotDotTest.zip Length Date Time Name --------- ---------- ----- ---- 7 07-26-2021 15:07 Hello.txt In other words, the `ZipFileSystem` doesn't end up creating a zip file which is then rejected by `ZipFileSystem` when creating a new filesystem using `Files.newFileSystem`. That's a good thing. ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From lancea at openjdk.java.net Mon Jul 26 10:16:47 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 26 Jul 2021 10:16:47 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v2] In-Reply-To: References: Message-ID: > Hi, > > As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS > > Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. > > > Mach5 tiers 1 through 3 have been run without any errors encountered . > > Best, > Lance Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: Add missing Copyright header and address minor comments ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4900/files - new: https://git.openjdk.java.net/jdk/pull/4900/files/68af64c4..2265ffe1 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=01 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=00-01 Stats: 25 lines in 2 files changed: 24 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/4900.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4900/head:pull/4900 PR: https://git.openjdk.java.net/jdk/pull/4900 From lancea at openjdk.java.net Mon Jul 26 10:16:53 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 26 Jul 2021 10:16:53 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v2] In-Reply-To: References: Message-ID: On Mon, 26 Jul 2021 07:30:12 GMT, Alan Bateman wrote: >> Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: >> >> Add missing Copyright header and address minor comments > > src/jdk.zipfs/share/classes/jdk/nio/zipfs/ZipFileSystem.java line 1573: > >> 1571: } >> 1572: IndexNode inode = new IndexNode(cen, pos, nlen); >> 1573: if(hasDotOrDotDot(inode.name)) { > > Minor nit, missing space in "if(". fixed > src/jdk.zipfs/share/classes/jdk/nio/zipfs/ZipFileSystem.java line 1602: > >> 1600: // Inode.name always includes "/" in path[0] >> 1601: if (path.length == 1) { >> 1602: return false; > > It may be useful to add "assert path[0] == '/';" at the start of this method. I added it per your suggestion, though IndexNode(byte[] cen, int pos, int nlen) which is only used by initCen() will already guarantee the leading "/" is there > test/jdk/jdk/nio/zipfs/HasDotDotTest.java line 1: > >> 1: import org.testng.annotations.DataProvider; > > Missing copyright header. Geez, how did I miss that. Added ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From lancea at openjdk.java.net Mon Jul 26 10:19:06 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 26 Jul 2021 10:19:06 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside In-Reply-To: References: Message-ID: On Mon, 26 Jul 2021 09:52:09 GMT, Jaikiran Pai wrote: > This change looks fine to me. I was unsure how the writing/creating entries with `.` or `..` with `ZipFileSystem` would behave in context of this change, so I gave this a try locally with the changes from this PR: > > ``` > try (FileSystem zipfs = FileSystems.newFileSystem(ZIPFILE)) { > final Path[] paths = new Path[]{ > zipfs.getPath("./Hello.txt"), > zipfs.getPath("../../../Hello.txt"), > zipfs.getPath("../Hello.txt")}; > for (int i = 0; i < paths.length; i++) { > try (OutputStream os = Files.newOutputStream(paths[i])) { > os.write(("Hello " + i).getBytes(StandardCharsets.UTF_8)); > } > } > } > ``` > > This code runs fine and it ends up creating (just one) CEN entry for `Hello.txt`: > > ``` > JTwork/scratch/zipfsDotDotTest.zip > Length Date Time Name > --------- ---------- ----- ---- > 7 07-26-2021 15:07 Hello.txt > ``` > > In other words, the `ZipFileSystem` doesn't end up creating a zip file which is then rejected by `ZipFileSystem` when creating a new filesystem using `Files.newFileSystem`. That's a good thing. With the exception of creating the Inodes table, Zip FS always calls ZipPath::getResolvedPath for access to Zip entries. So there is no issues with the creation and access of entries in the case above ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From alanb at openjdk.java.net Mon Jul 26 10:26:07 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Mon, 26 Jul 2021 10:26:07 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v2] In-Reply-To: References: Message-ID: On Mon, 26 Jul 2021 10:16:47 GMT, Lance Andersen wrote: >> Hi, >> >> As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS >> >> Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. >> >> >> Mach5 tiers 1 through 3 have been run without any errors encountered . >> >> Best, >> Lance > > Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: > > Add missing Copyright header and address minor comments > zipfs.getPath("./Hello.txt"), > zipfs.getPath("../../../Hello.txt"), > zipfs.getPath("../Hello.txt")}; > > > In other words, the `ZipFileSystem` doesn't end up creating a zip file which is then rejected by `ZipFileSystem` when creating a new filesystem using `Files.newFileSystem`. That's a good thing. Right, it's always existing behavior and matches the behavior of the platform file system. ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From paul.sandoz at oracle.com Mon Jul 26 15:54:13 2021 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Mon, 26 Jul 2021 15:54:13 +0000 Subject: Make java.nio.ByteBuffer a sealed class In-Reply-To: References: <11344009-DD3A-411F-93E5-0121DC95180D@oracle.com> Message-ID: <81996939-A032-4C61-8A35-AD007F8DCB4D@oracle.com> I understand the motivation, it's well intentioned, but I think there are better longer term approaches to reduce the technical debt by exploring use of the memory access and foreign API. ? It may be that with method patterns (something not yet proposed in a JEP) we could embed the knowledge and match, in totality, using ByteBuffer.isDirect(). Something to explore if/when such a pattern feature is available. Paul. > On Jul 23, 2021, at 2:48 PM, Sebastian Stenzel wrote: > > Sure, we can just assert that any ByteBuffer that isn't direct must be a HeapByteBuffer. But this assertion is based on our knowlege. The point is, that we could use a mechanism which would instead cause a compile time error if this ever changes. In my opinion this is the stronger tool to ensure correctness than relying on every contributor to know that only these existing implementations of ByteBuffers are allowed. > > This change has no direct functional effect, it merely helps to reduce branches that are technically dead code and just exist to satisfy ByteBuffer's contract. In other words: There is no functional advantage. This is about reducing technical debt. > >> Am 23.07.2021 um 17:46 schrieb Paul Sandoz : >> >> ?I don?t see much advantage to this. Most of the (byte) buffer hierarchy is package private. Further, the hierarchy is set up for convenience: DirectByteBuffer extends MappedByteBuffer [*], even though not all DirectByteBuffer instances are mapped. >> >> Internally we can assume if a ByteBuffer instance is also an instance sun.nio.ch.DirectBuffer (which in turn implies that ByteBuffer.isDirect() == true) then it?s a direct buffer. Otherwise, its a heap buffer and if hasArray() == true then the underlying array is accessible. >> >> In the referenced code there is a precondition that the byte buffer is not read-only. Therefore, we can assume for a heap buffer the src array is accessible (use an assert if you wish). >> >> My hope is this kind of code can improve when we can use MemorySegment. >> >> Paul. >> >> [*] >> public abstract class MappedByteBuffer >> extends ByteBuffer >> { >> >> // This is a little bit backwards: By rights MappedByteBuffer should be a >> // subclass of DirectByteBuffer, but to keep the spec clear and simple, and >> // for optimization purposes, it's easier to do it the other way around. >> // This works because DirectByteBuffer is a package-private class. >> >>> On Jul 23, 2021, at 3:59 AM, Sebastian Stenzel wrote: >>> >>> Any thoughts on this? >>> >>>>> On 8. Jul 2021, at 16:54, Sebastian Stenzel wrote: >>>> >>>> A while back, when working on JDK-8264110, we had a discussion about the nature of ByteBuffers, where I asked whether we could eventually make this a sealed class. [1] >>>> >>>> The idea is, that basically there are only two kinds of ByteBuffers: >>>> - HeapByteBuffers >>>> - MappedByteBuffer ? DirectByteBuffer >>>> >>>> While the former is backed by an array, the latter isn't, which leads to branching such as `if (buffer.hasArray())`, which could be simplified, if we knew for sure to deal with either of the two kinds. >>>> >>>> Now, that sealed classes are stable with JDK 17, I'd like to ask for your approval to make ByteBuffer a sealed class, so we can rely on an exhaustive list of ByteBuffer "flavours" at compile time. >>>> >>>> This is a requirement for follow-up tasks, I'd like to work on, such as JDK-8264341. >>>> >>>> >>>> [1] https://urldefense.com/v3/__https://github.com/openjdk/jdk/pull/3217*discussion_r602528233__;Iw!!ACWV5N9M2RV99hQ!d_JnBuyAJIsQrELaJHP-M0nHv3V2m3DSmNuEbXwf9IUGjQhCS_2YioIA2Vs5gTysvA$ >>> >> From sebastian.stenzel at gmail.com Mon Jul 26 16:07:42 2021 From: sebastian.stenzel at gmail.com (Sebastian Stenzel) Date: Mon, 26 Jul 2021 18:07:42 +0200 Subject: Make java.nio.ByteBuffer a sealed class In-Reply-To: <81996939-A032-4C61-8A35-AD007F8DCB4D@oracle.com> References: <81996939-A032-4C61-8A35-AD007F8DCB4D@oracle.com> Message-ID: I see, then I will add some asserts for now and remove the dead code and leave ByteBuffers untouched. :-) > Am 26.07.2021 um 17:54 schrieb Paul Sandoz : > > ?I understand the motivation, it's well intentioned, but I think there are better longer term approaches to reduce the technical debt by exploring use of the memory access and foreign API. > > ? > > It may be that with method patterns (something not yet proposed in a JEP) we could embed the knowledge and match, in totality, using ByteBuffer.isDirect(). Something to explore if/when such a pattern feature is available. > > Paul. > > >> On Jul 23, 2021, at 2:48 PM, Sebastian Stenzel wrote: >> >> Sure, we can just assert that any ByteBuffer that isn't direct must be a HeapByteBuffer. But this assertion is based on our knowlege. The point is, that we could use a mechanism which would instead cause a compile time error if this ever changes. In my opinion this is the stronger tool to ensure correctness than relying on every contributor to know that only these existing implementations of ByteBuffers are allowed. >> >> This change has no direct functional effect, it merely helps to reduce branches that are technically dead code and just exist to satisfy ByteBuffer's contract. In other words: There is no functional advantage. This is about reducing technical debt. >> >>>> Am 23.07.2021 um 17:46 schrieb Paul Sandoz : >>> >>> ?I don?t see much advantage to this. Most of the (byte) buffer hierarchy is package private. Further, the hierarchy is set up for convenience: DirectByteBuffer extends MappedByteBuffer [*], even though not all DirectByteBuffer instances are mapped. >>> >>> Internally we can assume if a ByteBuffer instance is also an instance sun.nio.ch.DirectBuffer (which in turn implies that ByteBuffer.isDirect() == true) then it?s a direct buffer. Otherwise, its a heap buffer and if hasArray() == true then the underlying array is accessible. >>> >>> In the referenced code there is a precondition that the byte buffer is not read-only. Therefore, we can assume for a heap buffer the src array is accessible (use an assert if you wish). >>> >>> My hope is this kind of code can improve when we can use MemorySegment. >>> >>> Paul. >>> >>> [*] >>> public abstract class MappedByteBuffer >>> extends ByteBuffer >>> { >>> >>> // This is a little bit backwards: By rights MappedByteBuffer should be a >>> // subclass of DirectByteBuffer, but to keep the spec clear and simple, and >>> // for optimization purposes, it's easier to do it the other way around. >>> // This works because DirectByteBuffer is a package-private class. >>> >>>> On Jul 23, 2021, at 3:59 AM, Sebastian Stenzel wrote: >>>> >>>> Any thoughts on this? >>>> >>>>>> On 8. Jul 2021, at 16:54, Sebastian Stenzel wrote: >>>>> >>>>> A while back, when working on JDK-8264110, we had a discussion about the nature of ByteBuffers, where I asked whether we could eventually make this a sealed class. [1] >>>>> >>>>> The idea is, that basically there are only two kinds of ByteBuffers: >>>>> - HeapByteBuffers >>>>> - MappedByteBuffer ? DirectByteBuffer >>>>> >>>>> While the former is backed by an array, the latter isn't, which leads to branching such as `if (buffer.hasArray())`, which could be simplified, if we knew for sure to deal with either of the two kinds. >>>>> >>>>> Now, that sealed classes are stable with JDK 17, I'd like to ask for your approval to make ByteBuffer a sealed class, so we can rely on an exhaustive list of ByteBuffer "flavours" at compile time. >>>>> >>>>> This is a requirement for follow-up tasks, I'd like to work on, such as JDK-8264341. >>>>> >>>>> >>>>> [1] https://urldefense.com/v3/__https://github.com/openjdk/jdk/pull/3217*discussion_r602528233__;Iw!!ACWV5N9M2RV99hQ!d_JnBuyAJIsQrELaJHP-M0nHv3V2m3DSmNuEbXwf9IUGjQhCS_2YioIA2Vs5gTysvA$ >>>> >>> > From github.com+1701815+mkarg at openjdk.java.net Mon Jul 26 18:02:29 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 26 Jul 2021 18:02:29 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v3] In-Reply-To: References: Message-ID: On Fri, 2 Jul 2021 06:20:29 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. Sorry for the delay. I now have pushed a first draft of the test. IIUC then five implementation have to be tested just for the actualy API, i. e. whether they throws NPE when `out` is `null`, and whether they transfer small, single turn, and multiple turn loads correctly. The tests are taken from `InputStream/TransferTo.java` as you proposed, and I am not using mocking but instead use `Channels.new*` as the existing tests you proposed as a blueprint. I hope I understood your requests and proposals correctly, if not please tell me. The tests pass well and proof that the new implementations work as expected and API-compliant. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Mon Jul 26 18:02:27 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 26 Jul 2021 18:02:27 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v4] In-Reply-To: References: Message-ID: <7PBopxz12bSrundwCfzqD7QBYqhVU39CYehUGIYGOwg=.632fc69f-2dfd-41df-aace-ad8965b2139b@github.com> > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has updated the pull request incrementally with four additional commits since the last revision: - Draft: Test for ChannelInputStream::transferTo - Draft: Using Channels API to invoke sun.nio.ch classes - Draft: MUST check params before ANY other operation - Draft: ChannelInputStream::transferTo Test ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/47ee00a2..b431fcd6 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=02-03 Stats: 229 lines in 2 files changed: 229 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Mon Jul 26 20:24:51 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 26 Jul 2021 20:24:51 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v5] In-Reply-To: References: Message-ID: <_ImTEjqgdig9vcnINDYEXqG2y0WYzY-fYJ8uOUlvnis=.6f8a51cc-12d9-4037-9078-0a3ebd71d4c3@github.com> > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: Draft: Test for ChannelInputStream::transferTo ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/b431fcd6..bfc699a0 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=03-04 Stats: 227 lines in 1 file changed: 0 ins; 0 del; 227 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Mon Jul 26 20:45:14 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Mon, 26 Jul 2021 20:45:14 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: Message-ID: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision: Draft: Test for ChannelInputStream::transferTo ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/bfc699a0..1f9dba3d Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=04-05 Stats: 173 lines in 1 file changed: 6 ins; 0 del; 167 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From lancea at openjdk.java.net Mon Jul 26 21:16:31 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Mon, 26 Jul 2021 21:16:31 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v11] In-Reply-To: <27fDIfBum2vSv7DmFs_PH8Z27NAYMVeMUmAa6OOJgfI=.b125d8c7-83a0-48d5-8342-d0da696fdbff@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> <2ttMZAIqSLz7XfsDi0CQjsaApF19Vk68hNNgh_WDIw0=.a4b275b9-3ce9-4101-9dfd-0db920386eb1@github.com> <27fDIfBum2vSv7DmFs_PH8Z27NAYMVeMUmAa6OOJgfI=.b125d8c7-83a0-48d5-8342-d0da696fdbff@github.com> Message-ID: <0hitV3isecJqQy6DhbOYBrBufoIj5I0xNINiPBzkbLA=.2fb39881-8a70-46b2-ad43-d0b090941667@github.com> On Fri, 23 Jul 2021 16:22:01 GMT, Lance Andersen wrote: > > Thank you for the review Alan. > > @LanceAndersen, I've run the tier1 tests locally with the latest PR and they have passed without any regressions. Given that we changed the implementation to wrap ByteArrayOutputStream instead of extending it, would you want to rerun some of your other tests that you had previously run, before I integrate this? > > Yes, I will run additional tests and report back after they complete I did not notice any new issues after running tier1 - tier3 ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From bpb at openjdk.java.net Tue Jul 27 00:02:39 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 27 Jul 2021 00:02:39 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Mon, 26 Jul 2021 20:45:14 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. src/java.base/share/classes/sun/nio/ch/ChannelInputStream.java line 179: > 177: for (long n = srcSize - srcPos; bytesWritten < n;) > 178: bytesWritten += src.transferTo(srcPos + bytesWritten, Long.MAX_VALUE, dest); > 179: return bytesWritten; If `src` is extended at an inconvenient point in time, for example between the return from a call to `src.transferTo()` that makes `bytesWritten < n` false and the evaluation of that terminating condition, then it appears that not all the content of `src` will be transferred to `dest`. I am not however sure that this violates any contract but it could be a behavioral change from the existing implementation. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From bpb at openjdk.java.net Tue Jul 27 01:00:38 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 27 Jul 2021 01:00:38 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Mon, 26 Jul 2021 20:45:14 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. src/java.base/share/classes/sun/nio/ch/ChannelOutputStream.java line 85: > 83: private byte[] bs; // Invoker's previous array > 84: private byte[] b1; > 85: It might be better to put the field declarations at the beginning of the class before the static methods. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From jpai at openjdk.java.net Tue Jul 27 01:57:52 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Tue, 27 Jul 2021 01:57:52 GMT Subject: RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v11] In-Reply-To: References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On Fri, 23 Jul 2021 14:58:01 GMT, Jaikiran Pai wrote: >> Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? >> >> The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. >> >> P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 > > Jaikiran Pai has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 15 additional commits since the last revision: > > - Merge latest from master branch > - Implement review suggestions: > - remove unnecessary "synchronized" > - no need to open the temp file twice > - remove no longer necessary javadoc comment on test > - review suggestion - wrap ByteArrayOutputStream instead of extending it > - reorganize the tests now that the temp file creation threshold isn't exposed as a user configurable value > - minor update to comment on FileRolloverOutputStream class > - remove no longer used constant > - use jtreg's construct of manual test > - Implement Alan's review suggestion - don't expose the threshold as a configuration property, instead set it internally to a specific value > - propagate back the original checked IOException to the callers > - ... and 5 more: https://git.openjdk.java.net/jdk/compare/f875aac8...991de6b9 Thank you for running the tests, Lance. ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From jpai at openjdk.java.net Tue Jul 27 02:00:39 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Tue, 27 Jul 2021 02:00:39 GMT Subject: Integrated: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream In-Reply-To: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> References: <0GJVLeCPoDI4FePU-Z-NMwJHaS2XyH9S28LgqI1LNB8=.e32743dc-a6dd-475d-88ab-16ed3f46beab@github.com> Message-ID: On Mon, 28 Jun 2021 03:41:20 GMT, Jaikiran Pai wrote: > Can I please get a review for this proposed fix for the issue reported in https://bugs.openjdk.java.net/browse/JDK-8190753? > > The commit here checks for the size of the zip entry before trying to create a `ByteArrayOutputStream` for that entry's content. A new jtreg test has been included in this commit to reproduce the issue and verify the fix. > > P.S: It's still a bit arguable whether it's a good idea to create the `ByteArrayOutputStream` with an initial size potentially as large as the `MAX_ARRAY_SIZE` or whether it's better to just use some smaller default value. However, I think that can be addressed separately while dealing with https://bugs.openjdk.java.net/browse/JDK-8011146 This pull request has now been integrated. Changeset: c3d8e922 Author: Jaikiran Pai URL: https://git.openjdk.java.net/jdk/commit/c3d8e9228d0558a2ce3e093c105c61ea7af2e1d1 Stats: 356 lines in 3 files changed: 350 ins; 0 del; 6 mod 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream Reviewed-by: lancea ------------- PR: https://git.openjdk.java.net/jdk/pull/4607 From bpb at openjdk.java.net Tue Jul 27 02:22:32 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Tue, 27 Jul 2021 02:22:32 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Mon, 26 Jul 2021 20:45:14 GMT, Markus KARG wrote: >> This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. >> >> As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: >> * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. >> * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. >> >> Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. >> >> I encourage everybody to discuss this draft: >> * Are there valid arguments for *not* doing this change? >> * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? >> * How to go on from here: What is missing to get this ready for an actual review? > > Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 67: > 65: test(readableByteChannelInput(), defaultOutput()); > 66: } > 67: This test looks like it's doing what it's supposed to do. I'm not asking to change it, but I think using TestNG might have given a simpler result with less work. For example, there could be one `DataProvider` supplying inputs which feed a `@Test` which expects an NPE, and another `DataProvider` supplying input-output pairs which feeds a `@Test` which validates the functionality. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From mik3hall at gmail.com Thu Jul 29 00:02:09 2021 From: mik3hall at gmail.com (Michael Hall) Date: Wed, 28 Jul 2021 19:02:09 -0500 Subject: New release DefaultFileSystemProviders Message-ID: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> Have changes been made to the new jdk17 and jdk18 releases for the jdk DefaultFileSystemProvider test? I was asked to verify a bug against the latest releases for DefaultFileSystemProvider related code. The code is not itself modular but includes some optional modular parts. It gets one error if run without any modular and a different one if run with these modular parts. That error with modular is? Error: LinkageError occurred while loading main class us.hall.hp.common.LoaderLaunchStub java.lang.ExceptionInInitializerError: null At first I thought this was a native link error but after looking it up it seems more like a serialization type error where some current class doesn?t match a prior one? So I am wondering if recent releases have made changes that make my older provider derived code incompatible? These would of probably been indicated by changes made to the jdk DefaulFileSystemProvider test case to keep that working? I don?t keep current with the latest development release sources or I could probably try comparing myself. If helpful verbose class loading that seems to lead up to the error on OS/X shows? [0.076s][info][class,load] jdk.internal.loader.NativeLibraries$NativeLibraryContext$$Lambda$5/0x000000080003c208 source: jdk.internal.loader.NativeLibraries [0.076s][info][class,load] java.util.ArrayDeque$DeqIterator source: jrt:/java.base [0.076s][info][class,load] jdk.internal.loader.NativeLibrary source: jrt:/java.base [0.076s][info][class,load] jdk.internal.loader.NativeLibraries$NativeLibraryImpl source: jrt:/java.base [0.077s][info][class,load] jdk.internal.loader.NativeLibraries$$Lambda$6/0x000000080003caa0 source: jdk.internal.loader.NativeLibraries [0.078s][info][class,load] java.util.concurrent.ConcurrentHashMap$ValuesView source: jrt:/java.base [0.078s][info][class,load] java.util.Enumeration source: jrt:/java.base [0.078s][info][class,load] java.util.concurrent.ConcurrentHashMap$Traverser source: jrt:/java.base [0.078s][info][class,load] java.util.concurrent.ConcurrentHashMap$BaseIterator source: jrt:/java.base [0.078s][info][class,load] java.util.concurrent.ConcurrentHashMap$ValueIterator source: jrt:/java.base [0.078s][info][class,load] java.nio.file.attribute.BasicFileAttributes source: jrt:/java.base [0.078s][info][class,load] java.nio.file.attribute.PosixFileAttributes source: jrt:/java.base [0.078s][info][class,load] sun.nio.fs.UnixFileAttributes source: jrt:/java.base [0.078s][info][class,load] sun.nio.fs.UnixFileStoreAttributes source: jrt:/java.base [0.078s][info][class,load] sun.nio.fs.UnixMountEntry source: jrt:/java.base [0.078s][info][class,load] jdk.internal.loader.BuiltinClassLoader$5 source: jrt:/java.base [0.079s][info][class,load] java.lang.module.ModuleReader source: jrt:/java.base [0.079s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemModuleReader source: jrt:/java.base [0.079s][info][class,load] jdk.internal.module.ModulePatcher$PatchedModuleReader source: jrt:/java.base [0.079s][info][class,load] jdk.internal.module.SystemModuleFinders$SystemImage source: jrt:/java.base [0.079s][info][class,load] jdk.internal.jimage.ImageReaderFactory source: jrt:/java.base [0.079s][info][class,load] java.nio.file.Paths source: jrt:/java.base [0.079s][info][class,load] java.lang.ExceptionInInitializerError source: jrt:/java.base From this it still almost seems to indicate a problem in loading a native library but I commented out the code that does that in the main class and the error remained. From brian.burkhalter at oracle.com Thu Jul 29 00:28:55 2021 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 29 Jul 2021 00:28:55 +0000 Subject: New release DefaultFileSystemProviders In-Reply-To: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> Message-ID: <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> Do you mean the test SetDefaultProvider.java? Is so there was one change earlier this month [1] but it does not appear related to the problems you are seeing. [1] https://github.com/openjdk/jdk17/commit/4fc3180f75e1cea4ebd613f8253be205d95f830c#diff-1b5076b5b17cb0adeb60349df3d2a50387b14bb2af0406ce4c5d0a577281e97c On Jul 28, 2021, at 5:02 PM, Michael Hall > wrote: Have changes been made to the new jdk17 and jdk18 releases for the jdk DefaultFileSystemProvider test? I was asked to verify a bug against the latest releases for DefaultFileSystemProvider related code. The code is not itself modular but includes some optional modular parts. It gets one error if run without any modular and a different one if run with these modular parts. That error with modular is? Error: LinkageError occurred while loading main class us.hall.hp.common.LoaderLaunchStub java.lang.ExceptionInInitializerError: null -------------- next part -------------- An HTML attachment was scrubbed... URL: From mik3hall at gmail.com Thu Jul 29 00:36:13 2021 From: mik3hall at gmail.com (Michael Hall) Date: Wed, 28 Jul 2021 19:36:13 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> Message-ID: > On Jul 28, 2021, at 7:28 PM, Brian Burkhalter wrote: > > Do you mean the test SetDefaultProvider.java? Is so there was one change earlier this month [1] but it does not appear related to the problems you are seeing. > > [1] https://github.com/openjdk/jdk17/commit/4fc3180f75e1cea4ebd613f8253be205d95f830c#diff-1b5076b5b17cb0adeb60349df3d2a50387b14bb2af0406ce4c5d0a577281e97c Related to the right bug but as you say not seeming to be related to mine. I did verify that bug is gone for your test provider. This is running against that with my modular turned on. /usr/libexec/java_home --exec java -cp . -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=TestProvider --module-path mods --add-modules us.hall.trz.osx,org.openjdk.nashorn,java.compiler,java.desktop,java.logging,java.management,java.prefs,java.se,java.rmi,java.scripting,java.sql,java.xml,jdk.attach,jdk.jshell,jdk.crypto.ec,jdk.jdeps,jdk.jcmd --add-exports org.openjdk.nashorn/org.openjdk.nashorn.tools=ALL-UNNAMED org.test.Test WARNING: A command line option has enabled the Security Manager WARNING: The Security Manager is deprecated and will be removed in a future release TestProvider$TestFileSystem at 5b6f7412 TestProvider Runs. Something different about mine somewhere. With your link maybe I can compare my source against current on the browser. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mik3hall at gmail.com Thu Jul 29 00:46:13 2021 From: mik3hall at gmail.com (Michael Hall) Date: Wed, 28 Jul 2021 19:46:13 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> Message-ID: > On Jul 28, 2021, at 7:36 PM, Michael Hall wrote: > > > >> On Jul 28, 2021, at 7:28 PM, Brian Burkhalter > wrote: >> >> Do you mean the test SetDefaultProvider.java? Is so there was one change earlier this month [1] but it does not appear related to the problems you are seeing. >> >> [1] https://github.com/openjdk/jdk17/commit/4fc3180f75e1cea4ebd613f8253be205d95f830c#diff-1b5076b5b17cb0adeb60349df3d2a50387b14bb2af0406ce4c5d0a577281e97c > Related to the right bug but as you say not seeming to be related to mine. > > I did verify that bug is gone for your test provider. > > This is running against that with my modular turned on. > > /usr/libexec/java_home --exec java -cp . -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=TestProvider --module-path mods --add-modules us.hall.trz.osx,org.openjdk.nashorn,java.compiler,java.desktop,java.logging,java.management,java.prefs,java.se ,java.rmi,java.scripting,java.sql,java.xml,jdk.attach,jdk.jshell,jdk.crypto.ec,jdk.jdeps,jdk.jcmd --add-exports org.openjdk.nashorn/org.openjdk.nashorn.tools=ALL-UNNAMED org.test.Test > WARNING: A command line option has enabled the Security Manager > WARNING: The Security Manager is deprecated and will be removed in a future release > TestProvider$TestFileSystem at 5b6f7412 > TestProvider > > Runs. Something different about mine somewhere. > With your link maybe I can compare my source against current on the browser. > > Thanks. If of interest, change that test to mine. /usr/libexec/java_home --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider --module-path mods --add-modules us.hall.trz.osx,org.openjdk.nashorn,java.compiler,java.desktop,java.logging,java.management,java.prefs,java.se,java.rmi,java.scripting,java.sql,java.xml,jdk.attach,jdk.jshell,jdk.crypto.ec,jdk.jdeps,jdk.jcmd --add-exports org.openjdk.nashorn/org.openjdk.nashorn.tools=ALL-UNNAMED org.test.Test Error occurred during initialization of VM java.lang.Error: java.lang.NullPointerException: Cannot invoke "java.nio.file.FileSystem.getPath(String, String[])" because the return value of "java.nio.file.FileSystems.getDefault()" is null at java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(java.base at 18-ea/FileSystems.java:134) at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(java.base at 18-ea/FileSystems.java:103) at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(java.base at 18-ea/FileSystems.java:101) at java.security.AccessController.doPrivileged(java.base at 18-ea/AccessController.java:318) at java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(java.base at 18-ea/FileSystems.java:101) at java.nio.file.FileSystems$DefaultFileSystemHolder.(java.base at 18-ea/FileSystems.java:94) at java.nio.file.FileSystems.getDefault(java.base at 18-ea/FileSystems.java:183) at java.nio.file.Path.of(java.base at 18-ea/Path.java:147) at java.nio.file.Paths.get(java.base at 18-ea/Paths.java:69) at jdk.internal.jimage.ImageReaderFactory.(java.base at 18-ea/ImageReaderFactory.java:51) at jdk.internal.module.SystemModuleFinders$SystemImage.(java.base at 18-ea/SystemModuleFinders.java:385) at jdk.internal.module.SystemModuleFinders$SystemModuleReader.containsImageLocation(java.base at 18-ea/SystemModuleFinders.java:446) at jdk.internal.module.SystemModuleFinders$SystemModuleReader.find(java.base at 18-ea/SystemModuleFinders.java:457) at jdk.internal.loader.BuiltinClassLoader.findResource(java.base at 18-ea/BuiltinClassLoader.java:497) at jdk.internal.loader.BuiltinClassLoader.findResource(java.base at 18-ea/BuiltinClassLoader.java:277) at jdk.internal.loader.BootLoader.findResource(java.base at 18-ea/BootLoader.java:164) at java.lang.Class.getResource(java.base at 18-ea/Class.java:2925) at java.lang.System.implSetSecurityManager(java.base at 18-ea/System.java:418) at java.lang.System.initPhase3(java.base at 18-ea/System.java:2203) Caused by: java.lang.NullPointerException at java.io.File.toPath(java.base at 18-ea/File.java:2396) at java.util.zip.ZipFile$Source.get(java.base at 18-ea/ZipFile.java:1246) at java.util.zip.ZipFile$CleanableResource.(java.base at 18-ea/ZipFile.java:708) at java.util.zip.ZipFile.(java.base at 18-ea/ZipFile.java:242) at java.util.zip.ZipFile.(java.base at 18-ea/ZipFile.java:172) at java.util.jar.JarFile.(java.base at 18-ea/JarFile.java:347) at jdk.internal.module.ModuleReferences$JarModuleReader.newJarFile(java.base at 18-ea/ModuleReferences.java:230) at jdk.internal.module.ModuleReferences$JarModuleReader.(java.base at 18-ea/ModuleReferences.java:237) at jdk.internal.module.ModuleReferences.lambda$newJarModule$0(java.base at 18-ea/ModuleReferences.java:94) at jdk.internal.module.ModuleReferenceImpl.open(java.base at 18-ea/ModuleReferenceImpl.java:93) at jdk.internal.loader.BuiltinClassLoader$5.apply(java.base at 18-ea/BuiltinClassLoader.java:1023) at jdk.internal.loader.BuiltinClassLoader$5.apply(java.base at 18-ea/BuiltinClassLoader.java:1020) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(java.base at 18-ea/ConcurrentHashMap.java:1708) at jdk.internal.loader.BuiltinClassLoader.moduleReaderFor(java.base at 18-ea/BuiltinClassLoader.java:1031) at jdk.internal.loader.BuiltinClassLoader.defineClass(java.base at 18-ea/BuiltinClassLoader.java:793) at jdk.internal.loader.BuiltinClassLoader.findClassInModuleOrNull(java.base at 18-ea/BuiltinClassLoader.java:741) at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base at 18-ea/BuiltinClassLoader.java:665) at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base at 18-ea/BuiltinClassLoader.java:639) at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base at 18-ea/ClassLoaders.java:188) at java.lang.ClassLoader.loadClass(java.base at 18-ea/ClassLoader.java:520) at java.lang.Class.forName0(java.base at 18-ea/Native Method) at java.lang.Class.forName(java.base at 18-ea/Class.java:467) at java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(java.base at 18-ea/FileSystems.java:124) at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(java.base at 18-ea/FileSystems.java:103) at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(java.base at 18-ea/FileSystems.java:101) at java.security.AccessController.doPrivileged(java.base at 18-ea/AccessController.java:318) at java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(java.base at 18-ea/FileSystems.java:101) at java.nio.file.FileSystems$DefaultFileSystemHolder.(java.base at 18-ea/FileSystems.java:94) at java.nio.file.FileSystems.getDefault(java.base at 18-ea/FileSystems.java:183) at java.nio.file.Path.of(java.base at 18-ea/Path.java:147) at java.nio.file.Paths.get(java.base at 18-ea/Paths.java:69) at jdk.internal.jimage.ImageReaderFactory.(java.base at 18-ea/ImageReaderFactory.java:51) at jdk.internal.module.SystemModuleFinders$SystemImage.(java.base at 18-ea/SystemModuleFinders.java:385) at jdk.internal.module.SystemModuleFinders$SystemModuleReader.containsImageLocation(java.base at 18-ea/SystemModuleFinders.java:446) at jdk.internal.module.SystemModuleFinders$SystemModuleReader.find(java.base at 18-ea/SystemModuleFinders.java:457) at jdk.internal.loader.BuiltinClassLoader.findResource(java.base at 18-ea/BuiltinClassLoader.java:497) at jdk.internal.loader.BuiltinClassLoader.findResource(java.base at 18-ea/BuiltinClassLoader.java:277) at jdk.internal.loader.BootLoader.findResource(java.base at 18-ea/BootLoader.java:164) at java.lang.Class.getResource(java.base at 18-ea/Class.java:2925) at java.lang.System.implSetSecurityManager(java.base at 18-ea/System.java:418) at java.lang.System.initPhase3(java.base at 18-ea/System.java:2203) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Jul 29 07:52:58 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 29 Jul 2021 08:52:58 +0100 Subject: New release DefaultFileSystemProviders In-Reply-To: References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> Message-ID: <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> On 29/07/2021 01:46, Michael Hall wrote: > > > If of interest, change that test to mine. > > /usr/libexec/java_home --exec java -cp .:macnio2 > -Djava.security.manager -Djava.security.policy=all.policy > -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider > --module-path mods > --add-modules?us.hall.trz.osx,org.openjdk.nashorn,java.compiler,java.desktop,java.logging,java.management,java.prefs,java.se > ,java.rmi,java.scripting,java.sql,java.xml,jdk.attach,jdk.jshell,jdk.crypto.ec,jdk.jdeps,jdk.jcmd??--add-exports?org.openjdk.nashorn/org.openjdk.nashorn.tools=ALL-UNNAMED > org.test.Test > Error occurred during initialization of VM > java.lang.Error: java.lang.NullPointerException: Cannot invoke > "java.nio.file.FileSystem.getPath(String, String[])" because?the > return value of "java.nio.file.FileSystems.getDefault()" is null Thanks for the stack trace, I think I understand it. It's another bootstrapping issue that arises when starting with a security manager, replacing the default file system provider, and running with an application module path. So three planets need to align to create the conditions. So while the changes in JDK-8266345 are correct, it's not the complete fix and there is work beyond the PolicyFile change to be able to startup with this configuration. I'll create another bug to track this and we'll try again. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From github.com+1701815+mkarg at openjdk.java.net Thu Jul 29 08:11:33 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 29 Jul 2021 08:11:33 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Tue, 27 Jul 2021 00:57:02 GMT, Brian Burkhalter wrote: >> Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. > > src/java.base/share/classes/sun/nio/ch/ChannelOutputStream.java line 85: > >> 83: private byte[] bs; // Invoker's previous array >> 84: private byte[] b1; >> 85: > > It might be better to put the field declarations at the beginning of the class before the static methods. This is a question of style and personal likes. Code maintenance is much easier if variables are defined *near* to its first use, not *far away* (as in the Pascal or C language). If the reviewers want me to change it, I will do it. > test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 67: > >> 65: test(readableByteChannelInput(), defaultOutput()); >> 66: } >> 67: > > This test looks like it's doing what it's supposed to do. I'm not asking to change it, but I think using TestNG might have given a simpler result with less work. For example, there could be one `DataProvider` supplying inputs which feed a `@Test` which expects an NPE, and another `DataProvider` supplying input-output pairs which feeds a `@Test` which validates the functionality. No doubt, using modern tools would have spared me work, and indeed I would have chosen JUnit or TestNG if there would be a clear commitment to that tool *upfront*. For now, I see now use in reworking the code afterwards. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Thu Jul 29 08:15:32 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 29 Jul 2021 08:15:32 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Mon, 26 Jul 2021 23:59:05 GMT, Brian Burkhalter wrote: >> Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. > > src/java.base/share/classes/sun/nio/ch/ChannelInputStream.java line 179: > >> 177: for (long n = srcSize - srcPos; bytesWritten < n;) >> 178: bytesWritten += src.transferTo(srcPos + bytesWritten, Long.MAX_VALUE, dest); >> 179: return bytesWritten; > > If `src` is extended at an inconvenient point in time, for example between the return from a call to `src.transferTo()` that makes `bytesWritten < n` false and the evaluation of that terminating condition, then it appears that not all the content of `src` will be transferred to `dest`. I am not however sure that this violates any contract but it could be a behavioral change from the existing implementation. The JavaDocs in `InputStream::transferTo` *cleary* tell the caller that there is **no** guarantee of *any* specific behavior in that particular case: >The behavior for the case where the input and/or output stream is asynchronously closed, or the thread interrupted during the transfer, is highly input and output stream specific, and therefore not specified. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From mik3hall at gmail.com Thu Jul 29 09:36:24 2021 From: mik3hall at gmail.com (Michael Hall) Date: Thu, 29 Jul 2021 04:36:24 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> Message-ID: > On Jul 29, 2021, at 2:52 AM, Alan Bateman wrote: > > On 29/07/2021 01:46, Michael Hall wrote: >> >> >> If of interest, change that test to mine. >> >> /usr/libexec/java_home --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider --module-path mods --add-modules us.hall.trz.osx,org.openjdk.nashorn,java.compiler,java.desktop,java.logging,java.management,java.prefs,java.se ,java.rmi,java.scripting,java.sql,java.xml,jdk.attach,jdk.jshell,jdk.crypto.ec,jdk.jdeps,jdk.jcmd --add-exports org.openjdk.nashorn/org.openjdk.nashorn.tools=ALL-UNNAMED org.test.Test >> Error occurred during initialization of VM >> java.lang.Error: java.lang.NullPointerException: Cannot invoke "java.nio.file.FileSystem.getPath(String, String[])" because the return value of "java.nio.file.FileSystems.getDefault()" is null > > Thanks for the stack trace, I think I understand it. It's another bootstrapping issue that arises when starting with a security manager, replacing the default file system provider, and running with an application module path. So three planets need to align to create the conditions. So while the changes in JDK-8266345 are correct, it's not the complete fix and there is work beyond the PolicyFile change to be able to startup with this configuration. I'll create another bug to track this and we'll try again. > > -Alan I hadn?t thought to change to the prior test before to get a stack crawl. I was getting pretty sure Astrology was involved somehow. If I can get a trace for the non-modular launch I?ll post that too. I?m pretty sure it is different. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpb at openjdk.java.net Thu Jul 29 16:45:34 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 29 Jul 2021 16:45:34 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Thu, 29 Jul 2021 08:12:23 GMT, Markus KARG wrote: >> src/java.base/share/classes/sun/nio/ch/ChannelInputStream.java line 179: >> >>> 177: for (long n = srcSize - srcPos; bytesWritten < n;) >>> 178: bytesWritten += src.transferTo(srcPos + bytesWritten, Long.MAX_VALUE, dest); >>> 179: return bytesWritten; >> >> If `src` is extended at an inconvenient point in time, for example between the return from a call to `src.transferTo()` that makes `bytesWritten < n` false and the evaluation of that terminating condition, then it appears that not all the content of `src` will be transferred to `dest`. I am not however sure that this violates any contract but it could be a behavioral change from the existing implementation. > > The JavaDocs in `InputStream::transferTo` *cleary* tell the caller that there is **no** guarantee of *any* specific behavior in that particular case: >>The behavior for the case where the input and/or output stream is asynchronously closed, or the thread interrupted during the transfer, is highly input and output stream specific, and therefore not specified. Good point. >> src/java.base/share/classes/sun/nio/ch/ChannelOutputStream.java line 85: >> >>> 83: private byte[] bs; // Invoker's previous array >>> 84: private byte[] b1; >>> 85: >> >> It might be better to put the field declarations at the beginning of the class before the static methods. > > This is a question of style and personal likes. Code maintenance is much easier if variables are defined *near* to its first use, not *far away* (as in the Pascal or C language). If the reviewers want me to change it, I will do it. It's actually a matter of convention but I think it can remain as it is. >> test/jdk/sun/nio/ch/ChannelInputStream/TransferTo.java line 67: >> >>> 65: test(readableByteChannelInput(), defaultOutput()); >>> 66: } >>> 67: >> >> This test looks like it's doing what it's supposed to do. I'm not asking to change it, but I think using TestNG might have given a simpler result with less work. For example, there could be one `DataProvider` supplying inputs which feed a `@Test` which expects an NPE, and another `DataProvider` supplying input-output pairs which feeds a `@Test` which validates the functionality. > > No doubt, using modern tools would have spared me work, and indeed I would have chosen JUnit or TestNG if there would be a clear commitment to that tool *upfront*. For now, I see now use in reworking the code afterwards. No need to rework it now. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From lancea at openjdk.java.net Thu Jul 29 18:21:07 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Thu, 29 Jul 2021 18:21:07 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v3] In-Reply-To: References: Message-ID: > Hi, > > As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS > > Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. > > > Mach5 tiers 1 through 3 have been run without any errors encountered . > > Best, > Lance Lance Andersen has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains six additional commits since the last revision: - Add Impl Note to Zip FS module-info - Merge - Add missing Copyright header and address minor comments - Address missing linefeed after package name - Address overzelous intellij import update - Patch to address JDK-8251329 ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4900/files - new: https://git.openjdk.java.net/jdk/pull/4900/files/2265ffe1..22731ac3 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=02 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=01-02 Stats: 3049 lines in 76 files changed: 1357 ins; 359 del; 1333 mod Patch: https://git.openjdk.java.net/jdk/pull/4900.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4900/head:pull/4900 PR: https://git.openjdk.java.net/jdk/pull/4900 From github.com+1701815+mkarg at openjdk.java.net Thu Jul 29 20:00:34 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Thu, 29 Jul 2021 20:00:34 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Thu, 29 Jul 2021 16:42:35 GMT, Brian Burkhalter wrote: >> The JavaDocs in `InputStream::transferTo` *cleary* tell the caller that there is **no** guarantee of *any* specific behavior in that particular case: >>>The behavior for the case where the input and/or output stream is asynchronously closed, or the thread interrupted during the transfer, is highly input and output stream specific, and therefore not specified. > > Good point. :-) >> This is a question of style and personal likes. Code maintenance is much easier if variables are defined *near* to its first use, not *far away* (as in the Pascal or C language). If the reviewers want me to change it, I will do it. > > It's actually a matter of convention but I think it can remain as it is. Ok for me, otherwise just clearly tell me and I do change it. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From naoto at openjdk.java.net Thu Jul 29 22:56:35 2021 From: naoto at openjdk.java.net (Naoto Sato) Date: Thu, 29 Jul 2021 22:56:35 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v3] In-Reply-To: References: Message-ID: <7zHatSac3UvRIlIuqeW4IvcvLIHAkc0P75O_gznJdlA=.e892404c-8063-4174-bb89-a5b295ac7507@github.com> On Thu, 29 Jul 2021 18:21:07 GMT, Lance Andersen wrote: >> Hi, >> >> As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS >> >> Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. >> >> >> Mach5 tiers 1 through 3 have been run without any errors encountered . >> >> Best, >> Lance > > Lance Andersen has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains six additional commits since the last revision: > > - Add Impl Note to Zip FS module-info > - Merge > - Add missing Copyright header and address minor comments > - Address missing linefeed after package name > - Address overzelous intellij import update > - Patch to address JDK-8251329 src/jdk.zipfs/share/classes/jdk/nio/zipfs/ZipFileSystem.java line 1622: > 1620: index++; > 1621: } > 1622: return dotOrDotDotFound; Could simply return `false` here, and eliminate the local variable `dotOrDotDotFound`. ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From mik3hall at gmail.com Thu Jul 29 23:00:24 2021 From: mik3hall at gmail.com (Michael Hall) Date: Thu, 29 Jul 2021 18:00:24 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> Message-ID: <73171134-7733-45C3-A258-A0F88152C352@gmail.com> > > I hadn?t thought to change to the prior test before to get a stack crawl. I was getting pretty sure Astrology was involved somehow. If I can get a trace for the non-modular launch I?ll post that too. I?m pretty sure it is different. Correction non-modular exploded works java 18 exploded works /usr/libexec/java_home -v 18 --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider org.test.Test WARNING: A command line option has enabled the Security Manager WARNING: The Security Manager is deprecated and will be removed in a future release us.hall.trz.osx.MacFileSystem at 60e53b93 us.hall.trz.osx.MacFileSystemProvider java 17 exploded works /usr/libexec/java_home -v 17 --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider org.test.Test WARNING: A command line option has enabled the Security Manager WARNING: The Security Manager is deprecated and will be removed in a future release us.hall.trz.osx.MacFileSystem at 511d50c0 us.hall.trz.osx.MacFileSystemProvider Non-exploded, or jar?d, is probably still https://bugs.openjdk.java.net/browse/JDK-8263940 From bpb at openjdk.java.net Thu Jul 29 23:17:46 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 29 Jul 2021 23:17:46 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call Message-ID: Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. ------------- Commit messages: - 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call Changes: https://git.openjdk.java.net/jdk/pull/4940/files Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4940&range=00 Issue: https://bugs.openjdk.java.net/browse/JDK-8271308 Stats: 197 lines in 4 files changed: 178 ins; 10 del; 9 mod Patch: https://git.openjdk.java.net/jdk/pull/4940.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4940/head:pull/4940 PR: https://git.openjdk.java.net/jdk/pull/4940 From bpb at openjdk.java.net Thu Jul 29 23:17:46 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 29 Jul 2021 23:17:46 GMT Subject: RFR: 8271308: (fc) FileChannel.transferTo() transfers no more than Integer.MAX_VALUE bytes in one call In-Reply-To: References: Message-ID: On Thu, 29 Jul 2021 23:08:58 GMT, Brian Burkhalter wrote: > Please consider this request to enhance `java.nio.channels.FileChannel.transferTo()` to be able to transfer more than `Integer.MAX_VALUE` bytes in one call. The approach taken avoids adding loops inside the various private `transferToX()` methods. Without this change the new test fails. No existing tests fail because of the change. The native method `maxTransferSize0()` might be better replaced by a constant private static final long MAX_TRANSFER_SIZE = (long)Integer.MAX_VALUE - 1; This would guarantee that Windows? `TransmitFile()` would not reject a transfer as being too large (`count == Integer.MAX_VALUE`) thereby forcing a slower path and would little affect the other platforms. ------------- PR: https://git.openjdk.java.net/jdk/pull/4940 From bpb at openjdk.java.net Thu Jul 29 23:42:41 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Thu, 29 Jul 2021 23:42:41 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: <1llS7FL5O9A3depy1oQWVTEBSuRQ1iKbJrhQUcEgREY=.020798ab-c8ce-40a1-804e-1112f51f6a44@github.com> On Thu, 29 Jul 2021 19:56:40 GMT, Markus KARG wrote: >> It's actually a matter of convention but I think it can remain as it is. > > Ok for me, otherwise just clearly tell me and I do change it. I think you can leave it unless someone else thinks otherwise. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From lancea at openjdk.java.net Fri Jul 30 00:55:57 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Fri, 30 Jul 2021 00:55:57 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v4] In-Reply-To: References: Message-ID: <_HLwah9MHEE00YhzFYmKEg38WvW5atpQetuzzv6qtbE=.6614b555-0b70-463c-aed1-2a6cc892089a@github.com> > Hi, > > As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS > > Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. > > > Mach5 tiers 1 through 3 have been run without any errors encountered . > > Best, > Lance Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: Remove uneeded variable ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4900/files - new: https://git.openjdk.java.net/jdk/pull/4900/files/22731ac3..67c726af Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=03 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=02-03 Stats: 2 lines in 1 file changed: 0 ins; 1 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/4900.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4900/head:pull/4900 PR: https://git.openjdk.java.net/jdk/pull/4900 From jpai at openjdk.java.net Fri Jul 30 01:46:33 2021 From: jpai at openjdk.java.net (Jaikiran Pai) Date: Fri, 30 Jul 2021 01:46:33 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v3] In-Reply-To: References: Message-ID: On Thu, 29 Jul 2021 18:21:07 GMT, Lance Andersen wrote: >> Hi, >> >> As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS >> >> Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. >> >> >> Mach5 tiers 1 through 3 have been run without any errors encountered . >> >> Best, >> Lance > > Lance Andersen has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains six additional commits since the last revision: > > - Add Impl Note to Zip FS module-info > - Merge > - Add missing Copyright header and address minor comments > - Address missing linefeed after package name > - Address overzelous intellij import update > - Patch to address JDK-8251329 src/jdk.zipfs/share/classes/module-info.java line 49: > 47: * > 48: * @implNote The Zip File System will throw a ZipException when opening an > 49: * existing Zip file that contains Zip entries with "." or ".." in its name elements. Hello Lance, reading this sentence adds a bit of confusion since it uses the word "contains". Had I not known the implemenation details, this sentence would have made me think zip file with name elements of the form `foo.bar` or `hello..` would also be rejected since these name elements "contain" `.` or `..` Do you think we should change the word to something like "The Zip File System will throw a ZipException when opening an existing Zip file that has "." or ".." named entries"? ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From alanb at openjdk.java.net Fri Jul 30 11:27:30 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 30 Jul 2021 11:27:30 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v3] In-Reply-To: References: Message-ID: On Fri, 30 Jul 2021 01:43:56 GMT, Jaikiran Pai wrote: >> Lance Andersen has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains six additional commits since the last revision: >> >> - Add Impl Note to Zip FS module-info >> - Merge >> - Add missing Copyright header and address minor comments >> - Address missing linefeed after package name >> - Address overzelous intellij import update >> - Patch to address JDK-8251329 > > src/jdk.zipfs/share/classes/module-info.java line 49: > >> 47: * >> 48: * @implNote The Zip File System will throw a ZipException when opening an >> 49: * existing Zip file that contains Zip entries with "." or ".." in its name elements. > > Hello Lance, reading this sentence adds a bit of confusion since it uses the word "contains". Had I not known the implemenation details, this sentence would have made me think zip file with name elements of the form `foo.bar` or `hello..` would also be rejected since these name elements "contain" `.` or `..` > > Do you think we should change the word to something like "The Zip File System will throw a ZipException when opening an existing Zip file that has "." or ".." named entries"? I think the `@implNote` tag can be dropped here. For the text then maybe it could be tweaked to something like "The ZIP file system provider does not support opening an existing ZIP that contains entries with ...". ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From alanb at openjdk.java.net Fri Jul 30 12:12:30 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Fri, 30 Jul 2021 12:12:30 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Thu, 29 Jul 2021 19:57:30 GMT, Markus KARG wrote: >> Good point. > > :-) Is this loop correct for the case that the channel gets truncated? In that case transferTo will return 0 as no bytes will be transferred and I'm concerned this code will go into an infinite loop. Also can you can check that IllegalBlockingModeException is thrown for the case ch is a SelectableChannel configured non-blocking and the destination is a FileChannel? Just on naming, the existing channel implementations use "dst" for the destination and "wbc" (not "oc") for writable byte channels. Just mentioning it so that the new code can be kept consistent where possible. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From Alan.Bateman at oracle.com Fri Jul 30 14:36:53 2021 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 30 Jul 2021 15:36:53 +0100 Subject: New release DefaultFileSystemProviders In-Reply-To: <73171134-7733-45C3-A258-A0F88152C352@gmail.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> Message-ID: On 30/07/2021 00:00, Michael Hall wrote: > Correction non-modular exploded works > > java 18 exploded works > /usr/libexec/java_home -v 18 --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider org.test.Test > WARNING: A command line option has enabled the Security Manager > WARNING: The Security Manager is deprecated and will be removed in a future release > us.hall.trz.osx.MacFileSystem at 60e53b93 > us.hall.trz.osx.MacFileSystemProvider > > java 17 exploded works > /usr/libexec/java_home -v 17 --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider org.test.Test > WARNING: A command line option has enabled the Security Manager > WARNING: The Security Manager is deprecated and will be removed in a future release > us.hall.trz.osx.MacFileSystem at 511d50c0 > us.hall.trz.osx.MacFileSystemProvider > > Non-exploded, or jar?d, is probably still https://bugs.openjdk.java.net/browse/JDK-8263940 Yes, the exploded case should be okay. However, if java.nio.file.spi.DefaultFileSystemProvider specifies a class in a JAR file that is on the class path or module path then you'll run into this issue.? Setting the security manager on the command line will trigger this occur at VM startup. -Alan From mik3hall at gmail.com Fri Jul 30 15:32:05 2021 From: mik3hall at gmail.com (Michael Hall) Date: Fri, 30 Jul 2021 10:32:05 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> Message-ID: > On Jul 30, 2021, at 9:36 AM, Alan Bateman wrote: > > On 30/07/2021 00:00, Michael Hall wrote: >> Correction non-modular exploded works >> >> java 18 exploded works >> /usr/libexec/java_home -v 18 --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider org.test.Test >> WARNING: A command line option has enabled the Security Manager >> WARNING: The Security Manager is deprecated and will be removed in a future release >> us.hall.trz.osx.MacFileSystem at 60e53b93 >> us.hall.trz.osx.MacFileSystemProvider >> >> java 17 exploded works >> /usr/libexec/java_home -v 17 --exec java -cp .:macnio2 -Djava.security.manager -Djava.security.policy=all.policy -Djava.nio.file.spi.DefaultFileSystemProvider=us.hall.trz.osx.MacFileSystemProvider org.test.Test >> WARNING: A command line option has enabled the Security Manager >> WARNING: The Security Manager is deprecated and will be removed in a future release >> us.hall.trz.osx.MacFileSystem at 511d50c0 >> us.hall.trz.osx.MacFileSystemProvider >> >> Non-exploded, or jar?d, is probably still https://bugs.openjdk.java.net/browse/JDK-8263940 > Yes, the exploded case should be okay. However, if java.nio.file.spi.DefaultFileSystemProvider specifies a class in a JAR file that is on the class path or module path then you'll run into this issue. Setting the security manager on the command line will trigger this occur at VM startup. > -Alan Thanks for explaining the conditions. I will hold off on additional testing for now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lancea at openjdk.java.net Fri Jul 30 15:43:52 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Fri, 30 Jul 2021 15:43:52 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v5] In-Reply-To: References: Message-ID: > Hi, > > As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS > > Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. > > > Mach5 tiers 1 through 3 have been run without any errors encountered . > > Best, > Lance Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: Update note added to Zip FS module-info ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4900/files - new: https://git.openjdk.java.net/jdk/pull/4900/files/67c726af..0bf065ea Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=04 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=03-04 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.java.net/jdk/pull/4900.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4900/head:pull/4900 PR: https://git.openjdk.java.net/jdk/pull/4900 From lancea at openjdk.java.net Fri Jul 30 15:47:31 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Fri, 30 Jul 2021 15:47:31 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v3] In-Reply-To: References: Message-ID: On Fri, 30 Jul 2021 11:24:37 GMT, Alan Bateman wrote: >> src/jdk.zipfs/share/classes/module-info.java line 49: >> >>> 47: * >>> 48: * @implNote The Zip File System will throw a ZipException when opening an >>> 49: * existing Zip file that contains Zip entries with "." or ".." in its name elements. >> >> Hello Lance, reading this sentence adds a bit of confusion since it uses the word "contains". Had I not known the implemenation details, this sentence would have made me think zip file with name elements of the form `foo.bar` or `hello..` would also be rejected since these name elements "contain" `.` or `..` >> >> Do you think we should change the word to something like "The Zip File System will throw a ZipException when opening an existing Zip file that has "." or ".." named entries"? > > I think the `@implNote` tag can be dropped here. For the text then maybe it could be tweaked to something like "The ZIP file system provider does not support opening an existing ZIP that contains entries with ...". Updated the note removing `@impleNote` thought about pre-pending **Note:** to make the wording stick out but held off for now. Updated the Release note accordingly Jaikiran, I think the use of name elements is fine as we use similar phrasing in `Path::getName()` ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From naoto at openjdk.java.net Fri Jul 30 17:09:29 2021 From: naoto at openjdk.java.net (Naoto Sato) Date: Fri, 30 Jul 2021 17:09:29 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v5] In-Reply-To: References: Message-ID: On Fri, 30 Jul 2021 15:43:52 GMT, Lance Andersen wrote: >> Hi, >> >> As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS >> >> Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. >> >> >> Mach5 tiers 1 through 3 have been run without any errors encountered . >> >> Best, >> Lance > > Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: > > Update note added to Zip FS module-info Looks good to me. test/jdk/jdk/nio/zipfs/HasDotDotTest.java line 116: > 114: @Test(dataProvider = "checkForDotOrDotDotPaths") > 115: public void hasDotOrDotDotTest(String path) throws IOException { > 116: if(DEBUG) { Nit: space after `if` ------------- Marked as reviewed by naoto (Reviewer). PR: https://git.openjdk.java.net/jdk/pull/4900 From lancea at openjdk.java.net Fri Jul 30 17:22:59 2021 From: lancea at openjdk.java.net (Lance Andersen) Date: Fri, 30 Jul 2021 17:22:59 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v6] In-Reply-To: References: Message-ID: <14bj0f-zLqVMsHsMvKV96kj8VyhOHya_6eOHj2DUbnM=.8f96c9d0-d64b-47bc-9f97-0b5b3a037e16@github.com> > Hi, > > As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS > > Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. > > > Mach5 tiers 1 through 3 have been run without any errors encountered . > > Best, > Lance Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: Add missing space in if statement ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4900/files - new: https://git.openjdk.java.net/jdk/pull/4900/files/0bf065ea..292e698f Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=05 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4900&range=04-05 Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod Patch: https://git.openjdk.java.net/jdk/pull/4900.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4900/head:pull/4900 PR: https://git.openjdk.java.net/jdk/pull/4900 From mik3hall at gmail.com Sat Jul 31 00:21:19 2021 From: mik3hall at gmail.com (Michael Hall) Date: Fri, 30 Jul 2021 19:21:19 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> Message-ID: <22EC882F-8C1F-44C7-ACF2-4DA67DCF33D7@gmail.com> >>> >> Yes, the exploded case should be okay. However, if java.nio.file.spi.DefaultFileSystemProvider specifies a class in a JAR file that is on the class path or module path then you'll run into this issue. Setting the security manager on the command line will trigger this occur at VM startup. >> -Alan > > Thanks for explaining the conditions. I will hold off on additional testing for now. > If you could maybe post the bug report number so I can track when I might test again. Thanks again. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Sat Jul 31 00:40:31 2021 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Sat, 31 Jul 2021 00:40:31 +0000 Subject: New release DefaultFileSystemProviders In-Reply-To: <22EC882F-8C1F-44C7-ACF2-4DA67DCF33D7@gmail.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> <22EC882F-8C1F-44C7-ACF2-4DA67DCF33D7@gmail.com> Message-ID: <2C103735-02B8-4C0D-AA11-F0A22BD669AF@oracle.com> On Jul 30, 2021, at 5:21 PM, Michael Hall > wrote: Yes, the exploded case should be okay. However, if java.nio.file.spi.DefaultFileSystemProvider specifies a class in a JAR file that is on the class path or module path then you'll run into this issue. Setting the security manager on the command line will trigger this occur at VM startup. -Alan Thanks for explaining the conditions. I will hold off on additional testing for now. If you could maybe post the bug report number so I can track when I might test again. Thanks again. This is it, I think: https://bugs.openjdk.java.net/browse/JDK-8263940 Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From mik3hall at gmail.com Sat Jul 31 00:42:22 2021 From: mik3hall at gmail.com (Michael Hall) Date: Fri, 30 Jul 2021 19:42:22 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: <2C103735-02B8-4C0D-AA11-F0A22BD669AF@oracle.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> <22EC882F-8C1F-44C7-ACF2-4DA67DCF33D7@gmail.com> <2C103735-02B8-4C0D-AA11-F0A22BD669AF@oracle.com> Message-ID: <6468C0A8-DC38-4F1D-BFFC-66BC4C768AD2@gmail.com> > On Jul 30, 2021, at 7:40 PM, Brian Burkhalter wrote: > > >> On Jul 30, 2021, at 5:21 PM, Michael Hall > wrote: >> >>>> Yes, the exploded case should be okay. However, if java.nio.file.spi.DefaultFileSystemProvider specifies a class in a JAR file that is on the class path or module path then you'll run into this issue. Setting the security manager on the command line will trigger this occur at VM startup. >>>> -Alan >>> >>> Thanks for explaining the conditions. I will hold off on additional testing for now. >>> >> If you could maybe post the bug report number so I can track when I might test again. >> >> Thanks again. > > This is it, I think: https://bugs.openjdk.java.net/browse/JDK-8263940 > > Brian Looks correct. Thanks Brian. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mik3hall at gmail.com Sat Jul 31 00:47:13 2021 From: mik3hall at gmail.com (Michael Hall) Date: Fri, 30 Jul 2021 19:47:13 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: <6468C0A8-DC38-4F1D-BFFC-66BC4C768AD2@gmail.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> <22EC882F-8C1F-44C7-ACF2-4DA67DCF33D7@gmail.com> <2C103735-02B8-4C0D-AA11-F0A22BD669AF@oracle.com> <6468C0A8-DC38-4F1D-BFFC-66BC4C768AD2@gmail.com> Message-ID: <63C00619-F00E-44B8-BB52-BE29E9D674A5@gmail.com> > On Jul 30, 2021, at 7:42 PM, Michael Hall wrote: > > > >> On Jul 30, 2021, at 7:40 PM, Brian Burkhalter > wrote: >> >> >>> On Jul 30, 2021, at 5:21 PM, Michael Hall > wrote: >>> >>>>> Yes, the exploded case should be okay. However, if java.nio.file.spi.DefaultFileSystemProvider specifies a class in a JAR file that is on the class path or module path then you'll run into this issue. Setting the security manager on the command line will trigger this occur at VM startup. >>>>> -Alan >>>> >>>> Thanks for explaining the conditions. I will hold off on additional testing for now. >>>> >>> If you could maybe post the bug report number so I can track when I might test again. >>> >>> Thanks again. >> >> This is it, I think: https://bugs.openjdk.java.net/browse/JDK-8263940 >> >> Brian > > Looks correct. Thanks Brian. > Actually the stack trace looks somewhat different from what I previously posted but I assume duplicates the same issue. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mik3hall at gmail.com Sat Jul 31 00:53:03 2021 From: mik3hall at gmail.com (Michael Hall) Date: Fri, 30 Jul 2021 19:53:03 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: <63C00619-F00E-44B8-BB52-BE29E9D674A5@gmail.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> <22EC882F-8C1F-44C7-ACF2-4DA67DCF33D7@gmail.com> <2C103735-02B8-4C0D-AA11-F0A22BD669AF@oracle.com> <6468C0A8-DC38-4F1D-BFFC-66BC4C768AD2@gmail.com> <63C00619-F00E-44B8-BB52-BE29E9D674A5@gmail.com> Message-ID: <64FFD862-C5F9-47E0-A306-E1CCF42E9E16@gmail.com> > On Jul 30, 2021, at 7:47 PM, Michael Hall wrote: > > > >> On Jul 30, 2021, at 7:42 PM, Michael Hall > wrote: >> >> >> >>> On Jul 30, 2021, at 7:40 PM, Brian Burkhalter > wrote: >>> >>> >>>> On Jul 30, 2021, at 5:21 PM, Michael Hall > wrote: >>>> >>>>>> Yes, the exploded case should be okay. However, if java.nio.file.spi.DefaultFileSystemProvider specifies a class in a JAR file that is on the class path or module path then you'll run into this issue. Setting the security manager on the command line will trigger this occur at VM startup. >>>>>> -Alan >>>>> >>>>> Thanks for explaining the conditions. I will hold off on additional testing for now. >>>>> >>>> If you could maybe post the bug report number so I can track when I might test again. >>>> >>>> Thanks again. >>> >>> This is it, I think: https://bugs.openjdk.java.net/browse/JDK-8263940 >>> >>> Brian >> >> Looks correct. Thanks Brian. >> > Actually the stack trace looks somewhat different from what I previously posted but I assume duplicates the same issue. Yeah, should of noticed, that was the first bug report with a jar class path that could be worked around by including the provider in an exploded directory. That works at jdk12. There is no current workaround. Could still use a number so I can include the provider with a current version of the application when it is fixed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpb at openjdk.java.net Sat Jul 31 00:59:19 2021 From: bpb at openjdk.java.net (Brian Burkhalter) Date: Sat, 31 Jul 2021 00:59:19 GMT Subject: RFR: 8057113: (fs) Path should have a method to obtain the filename extension [v9] In-Reply-To: References: Message-ID: <8vwe-oQ5ilpMKuep4zSHycQisXMIFImfyE4hRVFvCS8=.c0fbf5b2-ccf3-46f4-bce1-6bb5fb3403e3@github.com> > Please review this proposed change to add a method `java.nio.file.Path.getExtension()`. This was initially discussed in the thread http://mail.openjdk.java.net/pipermail/nio-dev/2018-February/004716.html. This method would return the filename extension of the file name of the `Path`. The extension is defined to be the portion of the file name after the last dot `(?.?)`. > > The definitions of file extension for about fifteen platforms and languages were surveyed to try to find a reasonable compromise for the definition of extension. The most common definition was the last segment of the name including and after the last dot. The second definition omitted the last dot from the extension. Java-related platforms all exclude the last dot. (One divergent definition in the internal Java NIO method `AbstractFileTypeDetector.getExtension(String)` defines the extension as the part after the *first* dot.) > > All examined cases define the extension to be an empty string if it cannot be determined. None of these cases used `null` to represent an indeterminate extension. > > Little in the way of specifying behavior for special cases (consisting mainly of file names with one or more leading dots) was found. Most definitions concern themselves only with the last dot and what comes after it and ignore leading dots altogether. A few definitions ignore a leading dot at the zeroth character. The current proposal ignores a dot at character zero. > > The behavior of the proposed method for some example cases is as: > > > . -> > .. -> > .a.b -> b > ...... -> > .....a -> a > ....a.b -> b > ..foo -> foo > test.rb -> rb > a/b/d/test.rb -> rb > .a/b/d/test.rb -> rb > foo. -> > test -> > .profile -> > .profile.sh -> sh > ..foo -> foo > .....foo -> foo > .vimrc -> > test. -> > test.. -> > test... -> > foo.tar.gz -> gz > foo.bar. -> > image.jpg -> jpg > music.mp3 -> mp3 > video.mp4 -> mp4 > document.txt -> txt > foo.tar.gz -> gz > foo.bar. -> Brian Burkhalter has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 10 additional commits since the last revision: - 8057113: Optimistically update @since tag to 18 - Merge - 8057113: Change first sentence; change param name - 8057113: Change Path.getExtension() to accept a default return value in case the extension is indeterminate - 8057113: Tweak spec again, and @implSpec code - 8057113: Add @since tag - 8057113: Tweak first sentence of spec - 8057113: Handle getFileName() == null; revise spec - 8057113: Changes pursuant to PR conversation - 8057113: (fs) Path should have a method to obtain the filename extension ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/2319/files - new: https://git.openjdk.java.net/jdk/pull/2319/files/e380d09e..fa00aec7 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=2319&range=08 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=2319&range=07-08 Stats: 178951 lines in 5300 files changed: 113351 ins; 39684 del; 25916 mod Patch: https://git.openjdk.java.net/jdk/pull/2319.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/2319/head:pull/2319 PR: https://git.openjdk.java.net/jdk/pull/2319 From mik3hall at gmail.com Sat Jul 31 01:39:11 2021 From: mik3hall at gmail.com (Michael Hall) Date: Fri, 30 Jul 2021 20:39:11 -0500 Subject: New release DefaultFileSystemProviders In-Reply-To: <64FFD862-C5F9-47E0-A306-E1CCF42E9E16@gmail.com> References: <1F9AE755-5E4D-43E9-920E-36AA398BC0F6@gmail.com> <68B9ADCD-B7F7-46CC-94C9-F8BF46A89B35@oracle.com> <2da73a16-df41-26c0-2d2a-ab382951a75a@oracle.com> <73171134-7733-45C3-A258-A0F88152C352@gmail.com> <22EC882F-8C1F-44C7-ACF2-4DA67DCF33D7@gmail.com> <2C103735-02B8-4C0D-AA11-F0A22BD669AF@oracle.com> <6468C0A8-DC38-4F1D-BFFC-66BC4C768AD2@gmail.com> <63C00619-F00E-44B8-BB52-BE29E9D674A5@gmail.com> <64FFD862-C5F9-47E0-A306-E1CCF42E9E16@gmail.com> Message-ID: <20AB8EFB-0017-4827-B328-A8529824E192@gmail.com> > On Jul 30, 2021, at 7:53 PM, Michael Hall wrote: > > > >> On Jul 30, 2021, at 7:47 PM, Michael Hall > wrote: >> >> > > Yeah, should of noticed, that was the first bug report with a jar class path that could be worked around by including the provider in an exploded directory. That works at jdk12. There is no current workaround. > Could still use a number so I can include the provider with a current version of the application when it is fixed. > Looking at it again maybe the same bug report works? Would the workaround no longer being available if any zip at all is present be worth a comment? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alanb at openjdk.java.net Sat Jul 31 09:40:31 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Sat, 31 Jul 2021 09:40:31 GMT Subject: RFR: 8251329: (zipfs) Files.walkFileTree walks infinitely if zip has dir named "." inside [v6] In-Reply-To: <14bj0f-zLqVMsHsMvKV96kj8VyhOHya_6eOHj2DUbnM=.8f96c9d0-d64b-47bc-9f97-0b5b3a037e16@github.com> References: <14bj0f-zLqVMsHsMvKV96kj8VyhOHya_6eOHj2DUbnM=.8f96c9d0-d64b-47bc-9f97-0b5b3a037e16@github.com> Message-ID: On Fri, 30 Jul 2021 17:22:59 GMT, Lance Andersen wrote: >> Hi, >> >> As discussed in the https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-July/079621.html thread, this is the revised patch to address the use of '.' and '..' within Zip FS >> >> Zip FS needs to use "." and ".." as links to the current and parent directories and cannot reliably support entries that have "." and ".." as name elements. This patch updates Zip Fs to reject ZIP files that have entries in the CEN that can't be used for files in a file system. >> >> >> Mach5 tiers 1 through 3 have been run without any errors encountered . >> >> Best, >> Lance > > Lance Andersen has updated the pull request incrementally with one additional commit since the last revision: > > Add missing space in if statement Marked as reviewed by alanb (Reviewer). ------------- PR: https://git.openjdk.java.net/jdk/pull/4900 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 31 13:17:08 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 31 Jul 2021 13:17:08 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v7] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Draft: Renamed oc to wbc and dest to dst ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/1f9dba3d..f91d5422 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=06 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=05-06 Stats: 17 lines in 1 file changed: 0 ins; 0 del; 17 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 31 13:17:09 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 31 Jul 2021 13:17:09 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Fri, 30 Jul 2021 12:09:30 GMT, Alan Bateman wrote: > Just on naming, the existing channel implementations use "dst" for the destination and "wbc" (not "oc") for writable byte channels. Just mentioning it so that the new code can be kept consistent where possible. I have renamed `dest` to `dst` and `oc` to `wbc` in https://github.com/openjdk/jdk/pull/4263/commits/f91d5422c41e25410a81aad54a45b2d0e171305a. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 31 14:14:03 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 31 Jul 2021 14:14:03 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v8] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has updated the pull request incrementally with two additional commits since the last revision: - Draft: Renamed r to bytesRead - Draft: Prevent infinite loop after truncation ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/f91d5422..f3e37fe1 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=07 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=06-07 Stats: 16 lines in 1 file changed: 2 ins; 2 del; 12 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 31 14:38:04 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 31 Jul 2021 14:38:04 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v9] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains two new commits since the last revision: - Draft: Renamed r to bytesRead - Draft: Prevent infinite loop after truncation ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/f3e37fe1..e0724718 Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=08 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=07-08 Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 31 14:41:39 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 31 Jul 2021 14:41:39 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Sat, 31 Jul 2021 13:12:54 GMT, Markus KARG wrote: >> Is this loop correct for the case that the channel gets truncated? In that case transferTo will return 0 as no bytes will be transferred and I'm concerned this code will go into an infinite loop. >> >> Also can you can check that IllegalBlockingModeException is thrown for the case ch is a SelectableChannel configured non-blocking and the destination is a FileChannel? >> >> Just on naming, the existing channel implementations use "dst" for the destination and "wbc" (not "oc") for writable byte channels. Just mentioning it so that the new code can be kept consistent where possible. > >> Just on naming, the existing channel implementations use "dst" for the destination and "wbc" (not "oc") for writable byte channels. Just mentioning it so that the new code can be kept consistent where possible. > > I have renamed `dest` to `dst` and `oc` to `wbc` in https://github.com/openjdk/jdk/pull/4263/commits/f91d5422c41e25410a81aad54a45b2d0e171305a. > Is this loop correct for the case that the channel gets truncated? In that case transferTo will return 0 as no bytes will be transferred and I'm concerned this code will go into an infinite loop. Good catch, indeed missed this particular situation! The modified code found in https://github.com/openjdk/jdk/pull/4263/commits/4b501b205c6f1c48bbc82d7a154aed3fc41b1a20 should be safe from infinite loops, as it checks the actual file length in each iteration (even if this costs us one more `synchronized` per iteration). This will also improve the situation with concurrent *extension* of the file as outlined in: >If src is extended at an inconvenient point in time, for example between the return from a call to src.transferTo() that makes bytesWritten < n false and the evaluation of that terminating condition, then it appears that not all the content of src will be transferred to dest. I am not however sure that this violates any contract but it could be a behavioral change from the existing implementation. As `size()` is called in each iteration, it will pick up the appendend bytes. Still the situation exists that some bytes are *replaced* in the source file -- but that is actually out-of-scope of this PR...! ;-) ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 31 16:02:07 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 31 Jul 2021 16:02:07 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Sat, 31 Jul 2021 14:39:02 GMT, Markus KARG wrote: >>> Just on naming, the existing channel implementations use "dst" for the destination and "wbc" (not "oc") for writable byte channels. Just mentioning it so that the new code can be kept consistent where possible. >> >> I have renamed `dest` to `dst` and `oc` to `wbc` in https://github.com/openjdk/jdk/pull/4263/commits/f91d5422c41e25410a81aad54a45b2d0e171305a. > >> Is this loop correct for the case that the channel gets truncated? In that case transferTo will return 0 as no bytes will be transferred and I'm concerned this code will go into an infinite loop. > > Good catch, indeed missed this particular situation! > > The modified code found in https://github.com/openjdk/jdk/pull/4263/commits/4b501b205c6f1c48bbc82d7a154aed3fc41b1a20 should be safe from infinite loops, as it checks the actual file length in each iteration (even if this costs us one more `synchronized` per iteration). > > This will also improve the situation with concurrent *extension* of the file as outlined in: > >>If src is extended at an inconvenient point in time, for example between the return from a call to src.transferTo() that makes bytesWritten < n false and the evaluation of that terminating condition, then it appears that not all the content of src will be transferred to dest. I am not however sure that this violates any contract but it could be a behavioral change from the existing implementation. > > As `size()` is called in each iteration, it will pick up the appendend bytes. Still the situation exists that some bytes are *replaced* in the source file -- but that is actually out-of-scope of this PR...! ;-) > Also can you can check that IllegalBlockingModeException is thrown for the case ch is a SelectableChannel configured non-blocking and the destination is a FileChannel? Done in https://github.com/openjdk/jdk/pull/4263/commits/8e2889fd6138d8aa8e308a1cd2aea3546a7c43a7, but honestly I'd kindly like to ask for a brief explanation why this has to be done. What actual bad effect would happen if I do not throw? ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 31 16:02:05 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 31 Jul 2021 16:02:05 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v10] In-Reply-To: References: Message-ID: > This PR-*draft* is **work in progress** and an invitation to discuss a possible solution for issue [JDK-8265891](https://bugs.openjdk.java.net/browse/JDK-8265891). It is *not yet* intended for a final review. > > As proposed in JDK-8265891, this PR provides an implementation for `Channels.newInputStream().transferTo()` which provide superior performance compared to the current implementation. The changes are: > * Prevents transfers through the JVM heap as much as possibly by offloading to deeper levels via NIO, hence allowing the operating system to optimize the transfer. > * Using more JRE heap in the fallback case when no NIO is possible (still only KiBs, hence mostl ynegligible even on SBCs) to better perform on modern hardware / fast I/O devides. > > Using JMH I have benchmarked both, the original implementation and this implementation, and (depending on the used hardware and use case) performance change was approx. doubled performance. So this PoC proofs that it makes sense to finalize this work and turn it into an actual OpenJDK contribution. > > I encourage everybody to discuss this draft: > * Are there valid arguments for *not* doing this change? > * Is there a *better* way to improve performance of `Channels.newInputStream().transferTo()`? > * How to go on from here: What is missing to get this ready for an actual review? Markus KARG has updated the pull request incrementally with one additional commit since the last revision: Draft: Throwing IllegalBlockingModeException if src is SelectableChannel and dst is FileChannel ------------- Changes: - all: https://git.openjdk.java.net/jdk/pull/4263/files - new: https://git.openjdk.java.net/jdk/pull/4263/files/e0724718..8e2889fd Webrevs: - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=09 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=4263&range=08-09 Stats: 3 lines in 1 file changed: 3 ins; 0 del; 0 mod Patch: https://git.openjdk.java.net/jdk/pull/4263.diff Fetch: git fetch https://git.openjdk.java.net/jdk pull/4263/head:pull/4263 PR: https://git.openjdk.java.net/jdk/pull/4263 From alanb at openjdk.java.net Sat Jul 31 17:36:30 2021 From: alanb at openjdk.java.net (Alan Bateman) Date: Sat, 31 Jul 2021 17:36:30 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Sat, 31 Jul 2021 15:58:07 GMT, Markus KARG wrote: >>> Is this loop correct for the case that the channel gets truncated? In that case transferTo will return 0 as no bytes will be transferred and I'm concerned this code will go into an infinite loop. >> >> Good catch, indeed missed this particular situation! >> >> The modified code found in https://github.com/openjdk/jdk/pull/4263/commits/4b501b205c6f1c48bbc82d7a154aed3fc41b1a20 should be safe from infinite loops, as it checks the actual file length in each iteration (even if this costs us one more `synchronized` per iteration). >> >> This will also improve the situation with concurrent *extension* of the file as outlined in: >> >>>If src is extended at an inconvenient point in time, for example between the return from a call to src.transferTo() that makes bytesWritten < n false and the evaluation of that terminating condition, then it appears that not all the content of src will be transferred to dest. I am not however sure that this violates any contract but it could be a behavioral change from the existing implementation. >> >> As `size()` is called in each iteration, it will pick up the appendend bytes. Still the situation exists that some bytes are *replaced* in the source file -- but that is actually out-of-scope of this PR...! ;-) > >> Also can you can check that IllegalBlockingModeException is thrown for the case ch is a SelectableChannel configured non-blocking and the destination is a FileChannel? > > Done in https://github.com/openjdk/jdk/pull/4263/commits/8e2889fd6138d8aa8e308a1cd2aea3546a7c43a7, but honestly I'd kindly like to ask for a brief explanation why this has to be done. What actual bad effect would happen if I do not throw? > The modified code found in [4b501b2](https://github.com/openjdk/jdk/commit/4b501b205c6f1c48bbc82d7a154aed3fc41b1a20) should be safe from infinite loops, as it checks the actual file length in each iteration (even if this costs us one more `synchronized` per iteration). I need to look at it closely but I suspect this introduces a potential overflow. Also if output stream is backed by a SocketChannel configured non-blocking then FC::transferTo may return 0 so I assume there is a potential infinite loop there too. I suspect the eventually patch will need have to make use of the blockingLock to prevent the underlying channels from being changed to non-blocking during the transfer. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263 From github.com+1701815+mkarg at openjdk.java.net Sat Jul 31 21:23:32 2021 From: github.com+1701815+mkarg at openjdk.java.net (Markus KARG) Date: Sat, 31 Jul 2021 21:23:32 GMT Subject: RFR: 8265891: (ch) InputStream returned by Channels.newInputStream should override transferTo [v6] In-Reply-To: References: <5PpEoJT1JZoo75hz4HAQFkipRfXCr_7MyTE8g4yhhgc=.5eeefcfe-205b-4b02-a59b-bead8b410139@github.com> Message-ID: On Sat, 31 Jul 2021 17:33:50 GMT, Alan Bateman wrote: > I need to look at it closely but I suspect this introduces a potential overflow. Also if output stream is backed by a SocketChannel configured non-blocking then FC::transferTo may return 0 so I assume there is a potential infinite loop there too. I suspect the eventually patch will need have to make use of the blockingLock to prevent the underlying channels from being changed to non-blocking during the transfer. I need to confess that my NIO knowledge is not deep enough to follow you closely, so I trust on your advice how to go on from here. ------------- PR: https://git.openjdk.java.net/jdk/pull/4263