From christoph.langer at sap.com Mon Apr 1 13:38:24 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Mon, 1 Apr 2019 13:38:24 +0000 Subject: RFR 8213031: (zipfs) Add support for POSIX file permissions In-Reply-To: References: <7b513a34196141f595cd5d194fb9d8a2@sap.com> <46af4f9d-60f9-c60d-e6f1-8fb97df3ba2e@oracle.com> <5d28b0715d2041ff892a3c44e024f120@sap.com> <8e9231ff-b7d5-bc2f-2643-713f3c670a1d@oracle.com> <3aeba9a64a434a968fc1d82a44077745@sap.com> <953449f0913340f6a94ae3d83611fd92@sap.com> <9b3c9cfe-63e9-94ea-1af0-7ba9e2db52fd@oracle.com> Message-ID: Hi Alan, thanks for getting back on this. > One thing that is puzzling is that > ZipFileAttributeView/ZipFileAttributes extend > PosixFileAttributeView/PosixFileAttributes. I don't think that will work > because the "zip" view is supported by default whereas "posix" view > needs the file system to be created with enablePosixFileAttributes=true. Hm, when I was looking at it initially, I was also wondering if it would be cleaner either have a default ZipFileAttributeView/ZipFileAttributes implementation that doesn't extend Posix or an "Enhanced" ZipFileAttributeView/ZipFileAttributes that would extend Posix. But I saw in the implementation of ZipFileAttributeView::get that this is already the "gate" where the requester would get the ZipFileAttributeView implementation with the requested behavior set. So I was hoping that it'd be fine to handle the Posix extension this way. Do you really think that wouldn't work? Alternatively, I could explore a different class hierarchy for ZipFileAttributeView/ZipFileAttributes... > I saw your comment about naming the file permissions attribute > "storedPermissions" in the zip view but if the zip and posix view are > separated then I think it would be clearer to name it "permissions" in > the zip view. If code is using Files.getAttribute then it will need the > view name so they won't get mixed up. Let me think about it... > You asked about the fallback when defaultOwner/defaultGroup aren't set. > If getOwner fails with an IOException then I think that exception should > be propagated. The UOE case will fallback to the value of "user.name". I > think the fallback for the group owner should be the file owner rather > than " > "". The default.policy file will need to be updated to > grant jdk.zipfs RuntimePermission("accessUserInfo") so that you won't > need to deal with security exceptions. Sounds fine. I'll implement that. > As regards next steps then I think we need to agree the spec changes, as > in the the javadoc in module-info.java. Once we have the spec agreed > then the CSR can be updated and finalized. The code review can be done > in parallel of course. I think Lance is going to help review the changes. Ok, I guess this eventually boils down to agree upon the right way of doing the "permissions" attribute or is there something more related to the spec? Thanks Christoph From Alan.Bateman at oracle.com Tue Apr 2 09:53:35 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Apr 2019 10:53:35 +0100 Subject: RFR 8213031: (zipfs) Add support for POSIX file permissions In-Reply-To: References: <7b513a34196141f595cd5d194fb9d8a2@sap.com> <46af4f9d-60f9-c60d-e6f1-8fb97df3ba2e@oracle.com> <5d28b0715d2041ff892a3c44e024f120@sap.com> <8e9231ff-b7d5-bc2f-2643-713f3c670a1d@oracle.com> <3aeba9a64a434a968fc1d82a44077745@sap.com> <953449f0913340f6a94ae3d83611fd92@sap.com> <9b3c9cfe-63e9-94ea-1af0-7ba9e2db52fd@oracle.com> Message-ID: <84ec224a-e0d2-e843-310c-bbf41638f928@oracle.com> On 01/04/2019 14:38, Langer, Christoph wrote: > : > Hm, when I was looking at it initially, I was also wondering if it would be cleaner either have a default ZipFileAttributeView/ZipFileAttributes implementation that doesn't extend Posix or an "Enhanced" ZipFileAttributeView/ZipFileAttributes that would extend Posix. But I saw in the implementation of ZipFileAttributeView::get that this is already the "gate" where the requester would get the ZipFileAttributeView implementation with the requested behavior set. So I was hoping that it'd be fine to handle the Posix extension this way. Do you really think that wouldn't work? > > Alternatively, I could explore a different class hierarchy for ZipFileAttributeView/ZipFileAttributes... The issue with ZipFileAttributeView extending PosixFileAttributeView is that it change the attributes defined by the latter to optional, e.g. try readAttributes "zip:*" when enablePosixFileAttributes is enabled and disabled. I think it would be simpler, as least for the specification, to separate them. > : > >> As regards next steps then I think we need to agree the spec changes, as >> in the the javadoc in module-info.java. Once we have the spec agreed >> then the CSR can be updated and finalized. The code review can be done >> in parallel of course. I think Lance is going to help review the changes. > Ok, I guess this eventually boils down to agree upon the right way of doing the "permissions" attribute or is there something more related to the spec? > We'll need to do a bit of wordsmithing too. For example, the current draft has "It is possible to query a Path object .." when it should be saying "It is possible to query a file in a Zip file system ...".? This is all straight-forward, I think the main things now are to get agreement on the relationship between the different sets of attributes and the name of the permissions that the zip view supports. -Alan From Alan.Bateman at oracle.com Tue Apr 2 10:22:52 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Apr 2019 11:22:52 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> Message-ID: On 02/04/2019 10:12, Andrew Dinn wrote: > Could I please have reviews for the following enhancement: > > JIRA: https://bugs.openjdk.java.net/browse/JDK-8221397 > webrev: http://cr.openjdk.java.net/~adinn/8221397/webrev.00/ > This updates the description for NonReadableChannelException/NonWriteableChannelException, I assume you didn't mean to include that in the patch. A corner case to consider is whether a map mode can be the empty string, maybe IAE should be thrown for that case. Are you planning a test to go along with this? It can test that the map method throws UOE and it can test the MapMode constructor. -Alan From shade at redhat.com Tue Apr 2 11:58:09 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 2 Apr 2019 13:58:09 +0200 Subject: RFR (XS) 8221824: Build failure with MSVS 2013 after JDK-8218418 Message-ID: Bug: https://bugs.openjdk.java.net/browse/JDK-8221824 I understand Oracle is building with newer MSVS. I would like to still maintain the code buildable with older MSVS-es. Fix: diff -r 35794e8db61b src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c --- a/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c Tue Apr 02 10:04:35 2019 +0200 +++ b/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c Tue Apr 02 13:56:19 2019 +0200 @@ -1066,6 +1066,9 @@ // Allow creation of symbolic links when the process is not elevated. // Developer Mode must be enabled for this option to function, otherwise - // it will be ignored. - DWORD dwFlags = (DWORD)flags | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; + // it will be ignored. Check that symbol is available in current build SDK. + DWORD dwFlags = (DWORD)flags; +#ifdef SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE + dwFlags |= SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; +#endif // On Windows 64-bit this appears to succeed even when there are Testing: local Windows build, jdk-submit (running) -- Thanks, -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From thomas.stuefe at gmail.com Tue Apr 2 12:46:21 2019 From: thomas.stuefe at gmail.com (=?UTF-8?Q?Thomas_St=C3=BCfe?=) Date: Tue, 2 Apr 2019 14:46:21 +0200 Subject: RFR (XS) 8221824: Build failure with MSVS 2013 after JDK-8218418 In-Reply-To: References: Message-ID: Looks good. Thanks for fixing. --Thomas On Tue, Apr 2, 2019 at 1:58 PM Aleksey Shipilev wrote: > Bug: > https://bugs.openjdk.java.net/browse/JDK-8221824 > > I understand Oracle is building with newer MSVS. I would like to still > maintain the code buildable > with older MSVS-es. > > Fix: > > diff -r 35794e8db61b > src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c > --- a/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c > Tue Apr 02 10:04:35 2019 +0200 > +++ b/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c > Tue Apr 02 13:56:19 2019 +0200 > @@ -1066,6 +1066,9 @@ > // Allow creation of symbolic links when the process is not elevated. > // Developer Mode must be enabled for this option to function, > otherwise > - // it will be ignored. > - DWORD dwFlags = (DWORD)flags | > SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; > + // it will be ignored. Check that symbol is available in current > build SDK. > + DWORD dwFlags = (DWORD)flags; > +#ifdef SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE > + dwFlags |= SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; > +#endif > > // On Windows 64-bit this appears to succeed even when there are > > Testing: local Windows build, jdk-submit (running) > > -- > Thanks, > -Aleksey > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Apr 2 12:56:34 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Apr 2019 13:56:34 +0100 Subject: RFR (XS) 8221824: Build failure with MSVS 2013 after JDK-8218418 In-Reply-To: References: Message-ID: <017c0766-1242-49a4-d52c-1beea88fefd3@oracle.com> On 02/04/2019 12:58, Aleksey Shipilev wrote: > Bug: > https://bugs.openjdk.java.net/browse/JDK-8221824 > > I understand Oracle is building with newer MSVS. I would like to still maintain the code buildable > with older MSVS-es. > > Fix: > > diff -r 35794e8db61b src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c > --- a/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c Tue Apr 02 10:04:35 2019 +0200 > +++ b/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c Tue Apr 02 13:56:19 2019 +0200 > @@ -1066,6 +1066,9 @@ > // Allow creation of symbolic links when the process is not elevated. > // Developer Mode must be enabled for this option to function, otherwise > - // it will be ignored. > - DWORD dwFlags = (DWORD)flags | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; > + // it will be ignored. Check that symbol is available in current build SDK. > + DWORD dwFlags = (DWORD)flags; > +#ifdef SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE > + dwFlags |= SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; > +#endif > > // On Windows 64-bit this appears to succeed even when there are > > Testing: local Windows build, jdk-submit (running) > With this patch then you undo JDK-8218418 when building on an older SDK. Instead, I think you need to set SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE when it's not set (its value seems to be 0x2). Separately, I'm interested to hear from Brian is this flag is ignored on older versions of Windows. If it isn't then a OS version check will be needed. -Alan From shade at redhat.com Tue Apr 2 13:04:33 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 2 Apr 2019 15:04:33 +0200 Subject: RFR (XS) 8221824: Build failure with MSVS 2013 after JDK-8218418 In-Reply-To: <017c0766-1242-49a4-d52c-1beea88fefd3@oracle.com> References: <017c0766-1242-49a4-d52c-1beea88fefd3@oracle.com> Message-ID: <630fa1fd-5c88-63ae-fdb0-7df6dea7a12f@redhat.com> On 4/2/19 2:56 PM, Alan Bateman wrote: >> diff -r 35794e8db61b src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c >> --- a/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c? Tue Apr 02 10:04:35 2019 >> +0200 >> +++ b/src/java.base/windows/native/libnio/fs/WindowsNativeDispatcher.c? Tue Apr 02 13:56:19 2019 >> +0200 >> @@ -1066,6 +1066,9 @@ >> ????? // Allow creation of symbolic links when the process is not elevated. >> ????? // Developer Mode must be enabled for this option to function, otherwise >> -??? // it will be ignored. >> -??? DWORD dwFlags = (DWORD)flags | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; >> +??? // it will be ignored. Check that symbol is available in current build SDK. >> +??? DWORD dwFlags = (DWORD)flags; >> +#ifdef SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE >> +??? dwFlags |= SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; >> +#endif >> >> ????? // On Windows 64-bit this appears to succeed even when there are >> >> Testing: local Windows build, jdk-submit (running) >> > With this patch then you undo JDK-8218418 when building on an older SDK. Instead, I think you need > to set SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE when it's not set (its value seems to be 0x2). I don't think it is a good idea to define and pass in flag that is not defined by the toolchain/headers. I think we either do this fix, or we backout the entire changeset that currently breaks the builds. Yes, current fix reverts the orginal change, but _only_ for older SDKs. Follow-ups that make it right for everything would be appreciated. -- Thanks, -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From Alan.Bateman at oracle.com Tue Apr 2 13:32:18 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Apr 2019 14:32:18 +0100 Subject: RFR (XS) 8221824: Build failure with MSVS 2013 after JDK-8218418 In-Reply-To: <630fa1fd-5c88-63ae-fdb0-7df6dea7a12f@redhat.com> References: <017c0766-1242-49a4-d52c-1beea88fefd3@oracle.com> <630fa1fd-5c88-63ae-fdb0-7df6dea7a12f@redhat.com> Message-ID: <64a66f97-6ad2-c283-735a-5fd9a466e812@oracle.com> On 02/04/2019 14:04, Aleksey Shipilev wrote: > : > I don't think it is a good idea to define and pass in flag that is not defined by the > toolchain/headers. I think we either do this fix, or we backout the entire changeset that currently > breaks the builds. Yes, current fix reverts the orginal change, but _only_ for older SDKs. > Follow-ups that make it right for everything would be appreciated. > It would be nice if there were some agreement on build-dev on the minimum SDK version to use - as I mentioned in the JIRA, we're using VS 2017 for JDK 13 so it will be hard to not break environments that are using VS 2013. Unfortunately there is long history of using #ifndef and/or GetProcAddress to make use of newer features while still building with older SDKs. So this wouldn't be the first. I don't object to your patch of course, I'm a lot more concerned as to how Windows 7 and 8.x behave when passed a flag that they do not recognize (this is a run-time issue rather than a build issue of course). -Alan From shade at redhat.com Tue Apr 2 13:58:18 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Tue, 2 Apr 2019 15:58:18 +0200 Subject: RFR (XS) 8221824: Build failure with MSVS 2013 after JDK-8218418 In-Reply-To: <64a66f97-6ad2-c283-735a-5fd9a466e812@oracle.com> References: <017c0766-1242-49a4-d52c-1beea88fefd3@oracle.com> <630fa1fd-5c88-63ae-fdb0-7df6dea7a12f@redhat.com> <64a66f97-6ad2-c283-735a-5fd9a466e812@oracle.com> Message-ID: <37e21792-c88d-1f77-270e-39b1a8c7b604@redhat.com> On 4/2/19 3:32 PM, Alan Bateman wrote: > On 02/04/2019 14:04, Aleksey Shipilev wrote: >> I don't think it is a good idea to define and pass in flag that is not defined by the >> toolchain/headers. I think we either do this fix, or we backout the entire changeset that currently >> breaks the builds. Yes, current fix reverts the orginal change, but _only_ for older SDKs. >> Follow-ups that make it right for everything would be appreciated. >> > It would be nice if there were some agreement on build-dev on the minimum SDK version to use - as I > mentioned in the JIRA, we're using VS 2017 for JDK 13 so it will be hard to not break environments > that are using VS 2013. Yes, that's understood. It does not mean we cannot make effort to keep it buildable :) > Unfortunately there is long history of using #ifndef and/or GetProcAddress to make use of newer > features while still building with older SDKs. So this wouldn't be the first. I don't object to your > patch of course, Okay, thanks. jdk-submit okay'ed the patch, I am going to push it shortly. -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From Alan.Bateman at oracle.com Tue Apr 2 14:03:48 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 2 Apr 2019 15:03:48 +0100 Subject: RFR (XS) 8221824: Build failure with MSVS 2013 after JDK-8218418 In-Reply-To: <37e21792-c88d-1f77-270e-39b1a8c7b604@redhat.com> References: <017c0766-1242-49a4-d52c-1beea88fefd3@oracle.com> <630fa1fd-5c88-63ae-fdb0-7df6dea7a12f@redhat.com> <64a66f97-6ad2-c283-735a-5fd9a466e812@oracle.com> <37e21792-c88d-1f77-270e-39b1a8c7b604@redhat.com> Message-ID: <88281e7b-fb88-9496-c78d-dc069915b9f9@oracle.com> On 02/04/2019 14:58, Aleksey Shipilev wrote: > : > Yes, that's understood. It does not mean we cannot make effort to keep it buildable :) I would prefer if we just dropped support for building with this really old SDK as it creates a tax for everyone touching Windows specific code. > >> Unfortunately there is long history of using #ifndef and/or GetProcAddress to make use of newer >> features while still building with older SDKs. So this wouldn't be the first. I don't object to your >> patch of course, > Okay, thanks. jdk-submit okay'ed the patch, I am going to push it shortly. I think there need to be follow to only specify this flag when running on Windows 10 or later. -Alan From Alan.Bateman at oracle.com Thu Apr 4 19:47:45 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 4 Apr 2019 20:47:45 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> Message-ID: On 03/04/2019 10:10, Andrew Dinn wrote: > : > I still have two undecided points you might advise on: > > Does the javadoc for FileChannel.map and/or FileChannelImpl.map need > updating to record the possibility that UOE might be thrown? I don't think I understand your question about FileChannel.map as you've already got @throws UOE in the patch. FileChannelImpl.map is JDK internal/implementation so nothing to do there (beyond the implementation change that you already have). > > Do I need to update any other implementations of map to cater for the > possibility of user-defined map modes? > No, the there shouldn't be anywhere else that needs updating. I looked through webrev.01. The package description has a catch-all for NPE so you need to specify that in each method. In the constructor it would be better to check for length 0 before assigning name (and of course name.length will throw NPE so you get that check for free). The test looks okay although I would probably catch the specific exceptions rather than catching Exception and catching them later but it amounts to the same checking. -Alan. From brian.burkhalter at oracle.com Fri Apr 5 00:46:11 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 4 Apr 2019 17:46:11 -0700 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time Message-ID: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> Whether SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE can be selected at runtime is not necessarily feasible using only the major and minor version numbers of Windows. If the major version is 10 and the minor version is 0 then the build number is also needed. The necessary logic is if (majorVersion > 10 || (majorVersion == 10 && (minorVersion > 0 || buildNumber >= 14972))) { flags |= SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; } Unfortunately the build number is not stored at either the Java or native level. There appear to be at least four approaches to solve the problem. 1. Propagate the build number to the Java level and put the logic there [1]. 2. Handle the availability of the flag via exceptions in native code [2]. 3. Obtain the build number in the native level and put the logic there [patch not developed]. 4. Revert the modified code to the revision level just before the patch for JDK-8218418 was applied. The patch for option 3 would likely require a new getVersionAndBuild() native method derived from the property initialization Windows native code which could itself be refactored to use it. The patches for options 1 and 2 have been tested on Windows 7, Windows Server 2016 (build 14393), and Windows 10 with, on the latter two platforms, Developer Mode both off and on. In all cases both patches exhibited the same desired and expected behavior. In patch [1], I am aware that SystemProps.java line 235 is out of order, and that lines 930-941 in WindowsNativeDispatcher.java could be more efficient. Despite the foregoing investigation, however, the relative value of the SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE flag compared to the quantity of changes required in options 1-3, option 4 might be the best choice. Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8221852/webrev-option-1.00/ [2] http://cr.openjdk.java.net/~bpb/8221852/webrev-option-2.00/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adinn at redhat.com Fri Apr 5 09:24:46 2019 From: adinn at redhat.com (Andrew Dinn) Date: Fri, 5 Apr 2019 10:24:46 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> Message-ID: <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> Hi Alan, Thanks for the feedback. On 04/04/2019 20:47, Alan Bateman wrote: > On 03/04/2019 10:10, Andrew Dinn wrote: >> : >> I still have two undecided points you might advise on: >> >> Does the javadoc for FileChannel.map and/or FileChannelImpl.map need >> updating to record the possibility that UOE might be thrown? > I don't think I understand your question about FileChannel.map as you've > already got @throws UOE in the patch. Yes, indeed :-) > FileChannelImpl.map is JDK internal/implementation so nothing to do > there (beyond the implementation change that you already have). Ok, thanks. > I looked through webrev.01. The package description has a catch-all for > NPE so you need to specify that in each method. In the constructor it > would be better to check for length 0 before assigning name (and of > course name.length will throw NPE so you get that check for free). The > test looks okay although I would probably catch the specific exceptions > rather than catching Exception and catching them later but it amounts to > the same checking. Sorry, I am afraid I don't understand what you mean by "The package description has a catch-all for NPE so you need to specify that in each method." What is the "package description"? Which methods of which class does the term "each method" range over? I have removed the call to Objects.requireNonNull from the MapMode constructor and let the call to length generate the NPE. The assignment of field name now follows the length check. I'll await answers to the above questions before presenting another webrev. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Fri Apr 5 09:47:33 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 5 Apr 2019 10:47:33 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> Message-ID: <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> On 05/04/2019 10:24, Andrew Dinn wrote: > : > Sorry, I am afraid I don't understand what you mean by "The package > description has a catch-all for NPE so you need to specify that in each > method." > > What is the "package description"? package-info.java has the package description. Near the end you should see the statement "Unless otherwise noted, passing a null ...".?? There are several areas of API that have statements like this to avoid cluttering the javadoc of every constructor/method that can throw NPE when invokes with a null argument. So this is why you don't see "@throws NullPointerException" here. -Alan From adinn at redhat.com Fri Apr 5 10:50:30 2019 From: adinn at redhat.com (Andrew Dinn) Date: Fri, 5 Apr 2019 11:50:30 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> Message-ID: <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> On 05/04/2019 10:47, Alan Bateman wrote: > On 05/04/2019 10:24, Andrew Dinn wrote: >> : >> Sorry, I am afraid I don't understand what you mean by "The package >> description has a catch-all for NPE so you need to specify that in each >> method." >> >> What is the "package description"? > package-info.java has the package description. Near the end you should > see the statement "Unless otherwise noted, passing a null ...".?? There > are several areas of API that have statements like this to avoid > cluttering the javadoc of every constructor/method that can throw NPE > when invokes with a null argument. So this is why you don't see "@throws > NullPointerException" here. Ah ok, got it now. I think I was confused by a missing 'not' in your original response. I have now removed the @throws for NPE from the MapMode constructor javadoc as well as letting the call to length() perform the null check. new webrev: http://cr.openjdk.java.net/~adinn/8221397/webrev.00/ JIRA: https://bugs.openjdk.java.net/browse/JDK-8221397 Testing: Successfully runs updated MapTest Submit job in progress Is this ok to push (modulo submit returning ok) or do I need another review for you and/or some other reviewer? regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Fri Apr 5 14:22:41 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 5 Apr 2019 15:22:41 +0100 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> Message-ID: <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> On 05/04/2019 01:46, Brian Burkhalter wrote: > Whether SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE can be selected > at runtime is not necessarily feasible using only the major and minor > version numbers of Windows. If the major version is 10 and the minor > version is 0 then the build number is also needed. The necessary logic is > > if (majorVersion > 10 > ? ? || (majorVersion == 10 && (minorVersion > 0 || buildNumber >= > 14972))) { > ? ? flags |= SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE; > } > > Unfortunately the build number is not stored at either the Java or > native level. There appear to be at least four approaches to solve the > problem. I met Robert Scholte (Maven) at FOSDEM and one of the things be brought up is that Maven are looking for additional OS version info to put into the mvn -version output [1]. Your #1 solution adds a os.buildNumber property and maybe an alternative to look at is existing os.version to add another element. It's hard to know how robust existing code that parses the value is so there may be compatibility impact to extending it beyond two elements. In any case, the original intention for WindowsNativeDispatcher was that the native methods map as close as possible to one win32 call so that the native code is as minimal as possible. Once Panama is further along then it should make it straight forward to migrate that code. As regards the approach then WindowsFileSystem is the place where OS versions/features are probed. If you dust off a jdk8 clone then you'll see where it enables sym links and enumeration of data streams based on the OS version. That code was removed when support for running on ancient releases of Windows was dropped but it should give you ideas on where the OS version check should be. I have no objection to your #4, at least temporarily until you've found a good approach. As it stands then it's problematic to have it set in the native method as it fail on some versions of Windows -Alan [1] https://issues.apache.org/jira/browse/MNG-6587 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Apr 5 15:36:07 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 5 Apr 2019 08:36:07 -0700 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> Message-ID: > On Apr 5, 2019, at 7:22 AM, Alan Bateman wrote: > >> Unfortunately the build number is not stored at either the Java or native level. There appear to be at least four approaches to solve the problem. > I met Robert Scholte (Maven) at FOSDEM and one of the things be brought up is that Maven are looking for additional OS version info to put into the mvn -version output [1]. In [1] the example output of ?mvn -version? is 10.0.16299.904. If I run ?ver? in ?cmd? in my Windows 7 VM I get 6.1.7601 which I know to be majorVersion.minorVersion.buildNumber. I don?t know what ?904? in the example from [1] represents. > Your #1 solution adds a os.buildNumber property and maybe an alternative to look at is existing os.version to add another element. It's hard to know how robust existing code that parses the value is so there may be compatibility impact to extending it beyond two elements. That?s why I did not add it to os.version. Another alternative would be to add something like os.fullVersion instead of os.buildNumber. This might be useful on other operating systems as well. > In any case, the original intention for WindowsNativeDispatcher was that the native methods map as close as possible to one win32 call so that the native code is as minimal as possible. I know. > Once Panama is further along then it should make it straight forward to migrate that code. > > As regards the approach then WindowsFileSystem is the place where OS versions/features are probed. If you dust off a jdk8 clone then you'll see where it enables sym links and enumeration of data streams based on the OS version. That code was removed when support for running on ancient releases of Windows was dropped but it should give you ideas on where the OS version check should be. > > I have no objection to your #4, at least temporarily until you've found a good approach. My inclination is to roll back the code to its state prior to [2]. This could be done under the current issue 8221852 or a new issue. Work on another approach can proceed then as time permits. > As it stands then it's problematic to have it set in the native method as it fail on some versions of Windows I don?t see why but we don?t want it in the WindowsNativeDispatcher code anyway. Note that the change to WindowsConstants in my option 2 of the previous message was inadvertent and not required. Thanks, Brian > [1] https://issues.apache.org/jira/browse/MNG-6587 [2] http://hg.openjdk.java.net/jdk/jdk/rev/66185e52b979 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Fri Apr 5 22:47:44 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 5 Apr 2019 15:47:44 -0700 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> Message-ID: > On Apr 5, 2019, at 8:36 AM, Brian Burkhalter wrote: > >> I have no objection to your #4, at least temporarily until you've found a good approach. > > My inclination is to roll back the code to its state prior to [2]. This could be done under the current issue 8221852 or a new issue. Work on another approach can proceed then as time permits. Well, after all I put together a different patch [A]. This reverts the native file to the version before the patch for 8218418. It moves all logic of the previous option 2 out of native code and out of the dispatcher code. The main change is the addition of WindowsLinkSupport.CreateSymLink(). The use of WindowsNativeDispatcher.CreateSymbolicLink() is replaced by this method in WindowsFileCopy and WindowsFileSystemProvider. There is also no change applied or needed with respect to obtaining more information about the OS version (build number). Testing on Windows 7 shows the same behavior as before 8218418. If the approach seems reasonable I can verify it on the other previously tested platforms (CI testing is in progress). Thanks, Brian [A] http://cr.openjdk.java.net/~bpb/8221852/webrev-option-2.01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From shade at redhat.com Fri Apr 5 23:56:25 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Sat, 6 Apr 2019 01:56:25 +0200 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> Message-ID: <14416183-ee5d-fbaf-3512-ce08565175b8@redhat.com> On 4/6/19 12:47 AM, Brian Burkhalter wrote: > [A]?http://cr.openjdk.java.net/~bpb/8221852/webrev-option-2.01/ FWIW, this passes Windows builds for me. -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From Alan.Bateman at oracle.com Sat Apr 6 18:01:20 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 6 Apr 2019 19:01:20 +0100 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> Message-ID: On 05/04/2019 23:47, Brian Burkhalter wrote: > > Well, after all I put together a different patch [A]. This reverts the > native file to the version before the patch for?8218418. It moves all > logic of the previous option 2 out of native code and out of the > dispatcher code. The main change is the addition of > WindowsLinkSupport.CreateSymLink(). The use of > WindowsNativeDispatcher.CreateSymbolicLink() is replaced by this > method in WindowsFileCopy and WindowsFileSystemProvider. There is also > no change applied or needed with respect to obtaining more information > about the OS version (build number). > I think it will error prone/confusion to have CreateSymLink in one source file and CreateSymbolicLink in another, it might be simply e to just move the the NOT_HELD handling into WindowsNativeDispatcher.CreateSymbolicLink. More medium term, we need to bring back the OS version detection in WindowsFileSystem so that it can set flags as to what features are supported depending on the OS version at runtime. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Sat Apr 6 18:05:42 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sat, 6 Apr 2019 19:05:42 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> Message-ID: On 05/04/2019 11:50, Andrew Dinn wrote: > : > Ah ok, got it now. I think I was confused by a missing 'not' in your > original response. > > I have now removed the @throws for NPE from the MapMode constructor > javadoc as well as letting the call to length() perform the null check. > > new webrev: http://cr.openjdk.java.net/~adinn/8221397/webrev.00/ > JIRA: https://bugs.openjdk.java.net/browse/JDK-8221397 > This versions look, I assume you'll create the CSR and wait for it to be approved before pushing. -Alan From adinn at redhat.com Mon Apr 8 09:52:43 2019 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 8 Apr 2019 10:52:43 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> Message-ID: On 06/04/2019 19:05, Alan Bateman wrote: > On 05/04/2019 11:50, Andrew Dinn wrote: >> : >> Ah ok, got it now. I think I was confused by a missing 'not' in your >> original response. >> >> I have now removed the @throws for NPE from the MapMode constructor >> javadoc as well as letting the call to length() perform the null check. >> >> new webrev: http://cr.openjdk.java.net/~adinn/8221397/webrev.00/ >> ?????? JIRA: https://bugs.openjdk.java.net/browse/JDK-8221397 >> > This versions look, I assume you'll create the CSR and wait for it to be > approved before pushing. I created the following CSR https://bugs.openjdk.java.net/browse/JDK-8222107 Apologies if I got anything wrong (first time I have done this :-). Do I need to solicit reviews or does this automatically get dispatched to the relevant component/sub-component leads? regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Mon Apr 8 11:34:32 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 8 Apr 2019 12:34:32 +0100 Subject: NIO based SocketImpl to replace legacy PlainSocketImpl In-Reply-To: <179696de-0340-8b1e-b802-2959ad59858b@oracle.com> References: <3225f8f0-08f2-c9bb-6dc4-b7eb7cfc4aa4@oracle.com> <179696de-0340-8b1e-b802-2959ad59858b@oracle.com> Message-ID: <42f4dbae-0e46-0137-03b9-bea5ad2ef772@oracle.com> Just a quick summary on where we are on this effort. I've submitted the JEP [1] so it can be considered for the roadmap. There wasn't any additional feedback from the draft that I sent here in February. In the mean-time, the "preparatory" changes have been pushed to jdk/jdk: JDK-8220493: Prepare Socket/ServerSocket for alternative platform SocketImpl JDK-8221259: New tests for java.net.Socket to exercise long standing behavior JDK-8220738: (sc) Move ServerSocketChannelImpl remaining native method to Net JDK-8217500: (sc) Move SocketChannelImpl's remaining native methods to Net JDK_8221252: (sc) SocketChannel and its socket adaptor need to handle connection reset JDK-8217461: (sc) Add Net.available to return the number of bytes in the socket input buffer JDK-8217451: ExtendedSocketOptions should encapsulate support for SO_FLOW_SLA JDK-8219446: Specify behaviour of timeout accepting methods of Socket and ServerSocket if timeout is negative and the CSR [2] has been removed to "Provisional". Getting the enabling changes pushed in advance means the changes are significantly reduced and are mostly one class with the new implementation. A snapshot of the patch from the niosocketimpl-branch is here: ? http://cr.openjdk.java.net/~alanb/8221481/0/webrev/index.html If the JEP becomes a candidate then then changes will need Reviewers. We will also need to decide on the system property to configure the JDK to use the legacy implementation. At this time, you can run with -Djdk.net.usePlainSocketImpl=true to use the legacy PlainSocketImpl. It probably not wroth spending time on anything more sophisticated. In addition to the above patch, the niosocketimpl-branch has updates to several existing socket tests to run them a second time with -Djdk.net.usePlainSocketImpl. I left them out of the above snapshot to avoid clutter as it's a one-line to each test description. As documented in the JEP, the motive is to avoid the legacy implementation bit rotting before it is removed. The niosocketimpl-branch also has Sergey Kuksenko's updates to the socket micros which may make it to the main line in advance of the changes. -Alan. [1] http://openjdk.java.net/jeps/8218559 [2] https://bugs.openjdk.java.net/browse/JDK-8222028 From Alan.Bateman at oracle.com Mon Apr 8 13:59:20 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 8 Apr 2019 14:59:20 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> Message-ID: <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> On 08/04/2019 10:52, Andrew Dinn wrote: > : > I created the following CSR > > https://bugs.openjdk.java.net/browse/JDK-8222107 > > Apologies if I got anything wrong (first time I have done this :-). > > Do I need to solicit reviews or does this automatically get dispatched > to the relevant component/sub-component leads? Once you have a Reviewer then you can finalize it. One thing that we haven't discussed is equality. If someone invokes FileChannel.map with a map mode created with new MapMode("READ_WRITE") then it will throw UOE as it's not (reference) equal to MapMode.READ_WRITE. It's implied by the javadoc "One of the constants ..." but now that it is extensible then we may need a statement in the javadoc - do you want me to suggest something? -Alan From adinn at redhat.com Mon Apr 8 14:05:29 2019 From: adinn at redhat.com (Andrew Dinn) Date: Mon, 8 Apr 2019 15:05:29 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> Message-ID: On 08/04/2019 14:59, Alan Bateman wrote: > On 08/04/2019 10:52, Andrew Dinn wrote: >> : >> I created the following CSR >> >> ?? https://bugs.openjdk.java.net/browse/JDK-8222107 >> >> Apologies if I got anything wrong (first time I have done this :-). >> >> Do I need to solicit reviews or does this automatically get dispatched >> to the relevant component/sub-component leads? > Once you have a Reviewer then you can finalize it. Ok, reviews for the above CSR are officially requested ... > One thing that we haven't discussed is equality. If someone invokes > FileChannel.map with a map mode created with new MapMode("READ_WRITE") > then it will throw UOE as it's not (reference) equal to > MapMode.READ_WRITE. It's implied by the javadoc "One of the constants > ..." but now that it is extensible then we may need a statement in the > javadoc - do you want me to suggest something? Hmm, yes that's an interesting point. Of course, I'll take whatever suggestions you are willing to throw this way :-) regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Tue Apr 9 09:09:08 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 9 Apr 2019 10:09:08 +0100 Subject: RFR: 8222107 (CSR) Support implementation-defined Map Modes Message-ID: Could I please get a review for the following CSR. It relates to the change proposed in JDK-8221397 which enables FileChannel implementation-specific extensions to enum MapMode. https://bugs.openjdk.java.net/browse/JDK-8222107 Thanks. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Tue Apr 9 10:42:46 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 9 Apr 2019 11:42:46 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range Message-ID: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> Could I please get reviews for the following patch which overloads MappedByteBuffer.force to accept a start offset and length. JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.00 This new API method was conceived as a preliminary change for JEP 352 to allow selective writeback of NVRAM-backed buffers. However, it has been implemented to provide a similar capability for file-mapped byte buffers. The old brute-force API method, force(), continues to operate as before for file-mapped byte buffers. One detail that is worth highlighting is that for file-backed buffers the start address passed to the native method force0 is rounded down to a page boundary. This is needed for Unix implementations to ensure that the underlying msync system call does not throw an exception. I am not sure whether Windows imposes this same restriction. If not then it might be better to perform the rounding in the native code. Advice would be welcome. Testing: I have only tested the new functionality on Linux. Assistance testing this on other OSes would be appreciated. [n.b. it should be enough to build with the patch and run $ make test TEST=test/jdk/java/nio/channels/FileChannel/MapTest.java" ] Updated MapTest.java: passes on Linux Submit Test: in progress regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From daniel.fuchs at oracle.com Tue Apr 9 11:28:06 2019 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 9 Apr 2019 12:28:06 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> Message-ID: <658eb7f1-e5e4-77f7-6795-65c59f99d8f3@oracle.com> Hi Andrew, On 09/04/2019 11:42, Andrew Dinn wrote: > One detail that is worth highlighting is that for file-backed buffers > the start address passed to the native method force0 is rounded down to > a page boundary. This is needed for Unix implementations to ensure that > the underlying msync system call does not throw an exception. Maybe this should be highlighted in the API documentation too, possibly as a non-normative implementation detail - stating that an implementation is free to do this (e.g. in an @implNote). My reading of your current proposed specification is that `from` is 234 * The offset to the first byte in the buffer region that 235 * is to be written back to storage and well - if I'm not mistaken then it appears the implementation can write some bytes before `from`, and that would be observable if you compared the file before and after calling force, isn't it? best regards, -- daniel > JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 > webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.00 From adinn at redhat.com Tue Apr 9 12:37:52 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 9 Apr 2019 13:37:52 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <658eb7f1-e5e4-77f7-6795-65c59f99d8f3@oracle.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <658eb7f1-e5e4-77f7-6795-65c59f99d8f3@oracle.com> Message-ID: <952b0d73-f123-bfd7-1e05-1e2eacf3ccfc@redhat.com> Hi Daniel, Thanks for looking at this. On 09/04/2019 12:28, Daniel Fuchs wrote: > Hi Andrew, > > On 09/04/2019 11:42, Andrew Dinn wrote: >> One detail that is worth highlighting is that for file-backed buffers >> the start address passed to the native method force0 is rounded down to >> a page boundary. This is needed for Unix implementations to ensure that >> the underlying msync system call does not throw an exception. > > Maybe this should be highlighted in the API documentation too, > possibly as a non-normative implementation detail - stating > that an implementation is free to do this (e.g. in an > @implNote). > > My reading of your current proposed specification is that > `from` is > ?234????? *??????? The offset to the first byte in the buffer region that > ?235????? *??????? is to be written back to storage > > and well - if I'm not mistaken then it appears the implementation > can write some bytes before `from`, and that would be observable > if you compared the file before and after calling force, isn't it? There is no implication in the current documentation that a call to force will /only/ write back bytes in the affected region. However, I agree that it should be stated explicitly that this may happen. I am not sure that this needs to be mentioned in an implNote. It is of the nature of most memory-mapped storage devices that writeback has a minimum granularity well above byte level. Would you be ok with a correspondingly general caveat? For example, what if I changed the second paragraph in the commment to: *

If the file mapped into this buffer resides on a local * storage device then when this method returns it is guaranteed * that all changes made to the selected region of the buffer since * it was created, or since this method was last invoked, will have * been written to that device. The force operation is free to * write bytes that lie outside the specified region, for example * to ensure that data blocks of some device-specific granularity * are transferred in their entirety. * regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Tue Apr 9 12:47:56 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 9 Apr 2019 13:47:56 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> Message-ID: <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> On 08/04/2019 15:05, Andrew Dinn wrote: > : > Hmm, yes that's an interesting point. Of course, I'll take whatever > suggestions you are willing to throw this way :-) > What would you think about not promoting the constructor to public? The original motivation for that was to be able to create additional map modes in jdk.internal.misc.ExtendedMapMode but we can use shared secrets or other skullduggery to do that. The advantage is that we avoid creating expectation in the API that developers can create their own map modes. The rest of the solution doesn't change, it's just eliminating the ability to create modes beyond the standard and extended modes. -Alan From adinn at redhat.com Tue Apr 9 12:55:19 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 9 Apr 2019 13:55:19 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> Message-ID: <5c285408-e790-013e-4766-67aeac629fc0@redhat.com> On 09/04/2019 13:47, Alan Bateman wrote: > On 08/04/2019 15:05, Andrew Dinn wrote: >> : >> Hmm, yes that's an interesting point. Of course, I'll take whatever >> suggestions you are willing to throw this way :-) >> > What would you think about not promoting the constructor to public? The > original motivation for that was to be able to create additional map > modes in jdk.internal.misc.ExtendedMapMode but we can use shared secrets > or other skullduggery to do that. The advantage is that we avoid > creating expectation in the API that developers can create their own map > modes. The rest of the solution doesn't change, it's just eliminating > the ability to create modes beyond the standard and extended modes. I'm happy to skulldig if you provide the advice on how to do it (an Idiot's Guide to Trepanning?). It definitely sounds better to expose this in a controlled way rather than via a general API. Can I assume this will remove the need for a CSR? regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Tue Apr 9 13:03:10 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 9 Apr 2019 14:03:10 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <5c285408-e790-013e-4766-67aeac629fc0@redhat.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> <5c285408-e790-013e-4766-67aeac629fc0@redhat.com> Message-ID: On 09/04/2019 13:55, Andrew Dinn wrote: > : > I'm happy to skulldig if you provide the advice on how to do it (an > Idiot's Guide to Trepanning?). It definitely sounds better to expose > this in a controlled way rather than via a general API. > > Can I assume this will remove the need for a CSR? > The CSR is still needed because of the change to the FileChannel.map API (that part is easy of course). -Alan From daniel.fuchs at oracle.com Tue Apr 9 15:04:19 2019 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 9 Apr 2019 16:04:19 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <952b0d73-f123-bfd7-1e05-1e2eacf3ccfc@redhat.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <658eb7f1-e5e4-77f7-6795-65c59f99d8f3@oracle.com> <952b0d73-f123-bfd7-1e05-1e2eacf3ccfc@redhat.com> Message-ID: Hi Andrew, On 09/04/2019 13:37, Andrew Dinn wrote: > I am not sure that this needs to be mentioned in an implNote. It is of > the nature of most memory-mapped storage devices that writeback has a > minimum granularity well above byte level. Would you be ok with a > correspondingly general caveat? > > For example, what if I changed the second paragraph in the commment to: > > *

If the file mapped into this buffer resides on a local > * storage device then when this method returns it is guaranteed > * that all changes made to the selected region of the buffer since > * it was created, or since this method was last invoked, will have > * been written to that device. The force operation is free to > * write bytes that lie outside the specified region, for example > * to ensure that data blocks of some device-specific granularity > * are transferred in their entirety. > * > That reads fine, thanks! BTW: I haven't really looked at the implementation, I'm leaving that to the experts of the field :-) best regards, -- daniel From adinn at redhat.com Tue Apr 9 15:11:09 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 9 Apr 2019 16:11:09 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <658eb7f1-e5e4-77f7-6795-65c59f99d8f3@oracle.com> <952b0d73-f123-bfd7-1e05-1e2eacf3ccfc@redhat.com> Message-ID: <45c3d862-8b2f-fa56-05e7-206bc3f4f43c@redhat.com> On 09/04/2019 16:04, Daniel Fuchs wrote: > . . . > That reads fine, thanks! Good. Thanks for reviewing this. > BTW: I haven't really looked at the implementation, I'm leaving that to > the experts of the field :-) No problem. I await their expert judgement. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Tue Apr 9 16:02:02 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 9 Apr 2019 17:02:02 +0100 Subject: RFR: 8222107 (CSR) Support implementation-defined Map Modes In-Reply-To: References: Message-ID: <87c489e8-9ad2-e9ec-11d6-e4fb06190871@redhat.com> On 09/04/2019 10:09, Andrew Dinn wrote: > Could I please get a review for the following CSR. It relates to the > change proposed in JDK-8221397 which enables FileChannel > implementation-specific extensions to enum MapMode. > > https://bugs.openjdk.java.net/browse/JDK-8222107 I have updated this CSR to fully detail all the javadoc changes required in FileChannel.impl. Following Alan Bateman's latest review of the implementation patch there is no need to make the MapMode constructor public. So the only changes required for this CSR are to javadoc. Reviews would be welcome. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Tue Apr 9 16:02:17 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 9 Apr 2019 17:02:17 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> Message-ID: <27be3165-74f7-6d55-2d1e-57d5afd00c8d@redhat.com> In response to Alan's suggestion (included below) I have reverted the constructor for MapMode to private and will use behind the scenes access to construct the extended enum values. This change removes the need to modify the test in MapTest.java (no new API to exercise). It still requires javadoc changes in FileChannel.map and implementation changes in FileChannelImpl.java. Reveiews for the udpated webrev would be welcome. I'll post a separate request for review of the CSR changes. JIRA: https://bugs.openjdk.java.net/browse/JDK-8221397 new webrev: http://cr.openjdk.java.net/~adinn/8221397/webrev.03 Testing: Submit repo: in progress regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander On 09/04/2019 13:47, Alan Bateman wrote: > What would you think about not promoting the constructor to public? The > original motivation for that was to be able to create additional map > modes in jdk.internal.misc.ExtendedMapMode but we can use shared secrets > or other skullduggery to do that. The advantage is that we avoid > creating expectation in the API that developers can create their own map > modes. The rest of the solution doesn't change, it's just eliminating > the ability to create modes beyond the standard and extended modes. From brian.burkhalter at oracle.com Tue Apr 9 16:10:59 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 9 Apr 2019 09:10:59 -0700 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: <14416183-ee5d-fbaf-3512-ce08565175b8@redhat.com> References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> <14416183-ee5d-fbaf-3512-ce08565175b8@redhat.com> Message-ID: <6FE34949-1FD9-488B-A310-F9815F38494D@oracle.com> > On Apr 5, 2019, at 4:56 PM, Aleksey Shipilev wrote: > > On 4/6/19 12:47 AM, Brian Burkhalter wrote: >> [A] http://cr.openjdk.java.net/~bpb/8221852/webrev-option-2.01/ > > FWIW, this passes Windows builds for me. Thanks for checking. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Apr 9 16:40:45 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 9 Apr 2019 09:40:45 -0700 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> Message-ID: <1B2E8AB4-CCCF-458B-A996-96D6D255CF78@oracle.com> > On Apr 6, 2019, at 11:01 AM, Alan Bateman wrote: > > On 05/04/2019 23:47, Brian Burkhalter wrote: >> >> Well, after all I put together a different patch [A]. This reverts the native file to the version before the patch for 8218418. It moves all logic of the previous option 2 out of native code and out of the dispatcher code. The main change is the addition of WindowsLinkSupport.CreateSymLink(). The use of WindowsNativeDispatcher.CreateSymbolicLink() is replaced by this method in WindowsFileCopy and WindowsFileSystemProvider. There is also no change applied or needed with respect to obtaining more information about the OS version (build number). >> > I think it will error prone/confusion to have CreateSymLink in one source file and CreateSymbolicLink in another, it might be simply e to just move the the NOT_HELD handling into WindowsNativeDispatcher.CreateSymbolicLink. Done [1]. Desired behavior manually verified: * Windows 7 - ERROR_PRIVILEGE_NOT_HELD * Windows Server 2016 (build 14393) - ERROR_PRIVILEGE_NOT_HELD with Dev Mode on and off * Windows 10 - ERROR_PRIVILEGE_NOT_HELD with Dev Mode off, success with Dev Mode on CI tiered sanity tests succeeded. > More medium term, we need to bring back the OS version detection in WindowsFileSystem so that it can set flags as to what features are supported depending on the OS version at runtime. I concur that it might be a good idea to make more precise version information available in the Java layer for other purposes, but I am not convinced that using it to determine whether to set this particular flag is any better than the fail-over this patch implements. The build number 14972 as of which this flag is supposed to be supported is rather casually documented [2] and we have no means of actually verifying this in testing. I think the existing logic while ugly is more robust. The issue of course can be revisited at the appropriate time. Brian [1] http://cr.openjdk.java.net/~bpb/8221852/webrev-option-2.02/ [2] https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/#DXz6icKZOkEozgYR.97 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Apr 9 16:49:22 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 9 Apr 2019 17:49:22 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <27be3165-74f7-6d55-2d1e-57d5afd00c8d@redhat.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> <27be3165-74f7-6d55-2d1e-57d5afd00c8d@redhat.com> Message-ID: <2026d6f7-3133-c6b9-ea50-18984d624810@oracle.com> On 09/04/2019 17:02, Andrew Dinn wrote: > In response to Alan's suggestion (included below) I have reverted the > constructor for MapMode to private and will use behind the scenes access > to construct the extended enum values. > > This change removes the need to modify the test in MapTest.java (no new > API to exercise). > > It still requires javadoc changes in FileChannel.map and implementation > changes in FileChannelImpl.java. Reveiews for the udpated webrev would > be welcome. I'll post a separate request for review of the CSR changes. > > JIRA: https://bugs.openjdk.java.net/browse/JDK-8221397 > new webrev: http://cr.openjdk.java.net/~adinn/8221397/webrev.03 > I think this looks fine. I've added myself as Reviewer to the CSR too. -Alan From Alan.Bateman at oracle.com Tue Apr 9 16:56:58 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 9 Apr 2019 17:56:58 +0100 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: <1B2E8AB4-CCCF-458B-A996-96D6D255CF78@oracle.com> References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> <1B2E8AB4-CCCF-458B-A996-96D6D255CF78@oracle.com> Message-ID: <3c887a08-6c73-70a4-459b-e75fa7922a08@oracle.com> On 09/04/2019 17:40, Brian Burkhalter wrote: > : > > Done [1]. Desired behavior manually verified: > > * Windows 7 - ERROR_PRIVILEGE_NOT_HELD > * Windows Server 2016 (build 14393) - ERROR_PRIVILEGE_NOT_HELD with > Dev Mode on and off > * Windows 10 - ERROR_PRIVILEGE_NOT_HELD with Dev Mode off, success > with Dev Mode on > > CI tiered sanity tests succeeded. This version looks good. The only thing is that restores the comment "On Windows 64-bit this appears to succeed even when there is insufficient privileges". Is that accurate? I'm just wondering if it should be removed. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Apr 9 17:00:43 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 9 Apr 2019 10:00:43 -0700 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: <3c887a08-6c73-70a4-459b-e75fa7922a08@oracle.com> References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> <1B2E8AB4-CCCF-458B-A996-96D6D255CF78@oracle.com> <3c887a08-6c73-70a4-459b-e75fa7922a08@oracle.com> Message-ID: > On Apr 9, 2019, at 9:56 AM, Alan Bateman wrote: > > This version looks good. The only thing is that restores the comment "On Windows 64-bit this appears to succeed even when there is insufficient privileges". Is that accurate? I'm just wondering if it should be removed. Yeah that seems wrong. I can excise it before pushing. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Apr 9 17:59:30 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 9 Apr 2019 18:59:30 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> Message-ID: <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> On 09/04/2019 11:42, Andrew Dinn wrote: > Could I please get reviews for the following patch which overloads > MappedByteBuffer.force to accept a start offset and length. > > JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 > webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.00 > > This new API method was conceived as a preliminary change for JEP 352 to > allow selective writeback of NVRAM-backed buffers. However, it has been > implemented to provide a similar capability for file-mapped byte > buffers. There are a couple of implementation details to discuss but I'll stick to the API/javadoc for now so that you can create the CSR. We discussed the method signature, parameter types, and the semantics in previous mails so I think they are all okay (meaning the first parameter is the index in the buffer of the first byte to write back and not an offset from the position). One nit is that the existing absolute methods name the parameter "index" and I think we should do the same here. In the first sentence it might be better to say "a region" rather "some region". You could expand this paragraph with a second sentence, something like "The region starts at the given {@code index} in this buffer and is {@code length} bytes". A suggestion for the description for the index and length parameters is copy the wording/style from the existing absolute methods, e.g. @param index The index of the first byte of the regsion; must be non-negative and less than limit() @param length The length of the region in bytes; must be non-negative and no larger than limit() - index @throws IndexOutOfBoundException If the preconditions on the index and length do not hold. -Alan From brian.burkhalter at oracle.com Tue Apr 9 18:15:02 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 9 Apr 2019 11:15:02 -0700 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> Message-ID: <32C928B7-C744-425F-8F9B-A02CD3AFBCE1@oracle.com> > On Apr 9, 2019, at 10:59 AM, Alan Bateman wrote: > > There are a couple of implementation details to discuss I was wondering whether this topic has any synergy / overlap with this prior thread [1] the principal sticking point of which was IIRC forcing / loading a slice of a MBB. Brian http://mail.openjdk.java.net/pipermail/nio-dev/2019-February/005871.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Apr 9 18:19:33 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 9 Apr 2019 11:19:33 -0700 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> <1B2E8AB4-CCCF-458B-A996-96D6D255CF78@oracle.com> <3c887a08-6c73-70a4-459b-e75fa7922a08@oracle.com> Message-ID: <8FD6CE6E-212E-4551-8859-E25CE1A1FFBD@oracle.com> > On Apr 9, 2019, at 10:00 AM, Brian Burkhalter wrote: > >> On Apr 9, 2019, at 9:56 AM, Alan Bateman > wrote: >> >> This version looks good. The only thing is that restores the comment "On Windows 64-bit this appears to succeed even when there is insufficient privileges". Is that accurate? I'm just wondering if it should be removed. > > Yeah that seems wrong. I can excise it before pushing. Webrev so updated [1]. Brian [1] http://cr.openjdk.java.net/~bpb/8221852/webrev-option-2.03/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Apr 9 19:16:01 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 9 Apr 2019 20:16:01 +0100 Subject: 8221852: SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE should be selected at runtime, not build time In-Reply-To: <8FD6CE6E-212E-4551-8859-E25CE1A1FFBD@oracle.com> References: <18405307-AE3D-464F-A080-86AB83A778D6@oracle.com> <9eed502e-7531-f2b7-ae3e-a9a155cd2309@oracle.com> <1B2E8AB4-CCCF-458B-A996-96D6D255CF78@oracle.com> <3c887a08-6c73-70a4-459b-e75fa7922a08@oracle.com> <8FD6CE6E-212E-4551-8859-E25CE1A1FFBD@oracle.com> Message-ID: <7a90243a-2787-8388-cdcd-25665b972aa4@oracle.com> On 09/04/2019 19:19, Brian Burkhalter wrote: > : > > Webrev so updated [1]. > > Brian > > [1] http://cr.openjdk.java.net/~bpb/8221852/webrev-option-2.03/ Looks good. -Alan From adinn at redhat.com Wed Apr 10 08:34:10 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 10 Apr 2019 09:34:10 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <2026d6f7-3133-c6b9-ea50-18984d624810@oracle.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> <27be3165-74f7-6d55-2d1e-57d5afd00c8d@redhat.com> <2026d6f7-3133-c6b9-ea50-18984d624810@oracle.com> Message-ID: <98be0420-303e-3b4a-ed83-dc75996fa0a1@redhat.com> On 09/04/2019 17:49, Alan Bateman wrote: > On 09/04/2019 17:02, Andrew Dinn wrote: >> In response to Alan's suggestion (included below) I have reverted the >> constructor for MapMode to private and will use behind the scenes access >> to construct the extended enum values. >> >> This change removes the need to modify the test in MapTest.java (no new >> API to exercise). >> >> It still requires javadoc changes in FileChannel.map and implementation >> changes in FileChannelImpl.java. Reveiews for the udpated webrev would >> be welcome. I'll post a separate request for review of the CSR changes. >> >> JIRA:?????? https://bugs.openjdk.java.net/browse/JDK-8221397 >> new webrev: http://cr.openjdk.java.net/~adinn/8221397/webrev.03 >> > I think this looks fine. I've added myself as Reviewer to the CSR too. Thank you, Alan. Does that mean it is ok to push this now or do I need a second reviewer? regards, Andrew Dinn ----------- From adinn at redhat.com Wed Apr 10 08:41:27 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 10 Apr 2019 09:41:27 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <81b955f9-b493-cc70-3bd1-c9b8c074739b@redhat.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <81b955f9-b493-cc70-3bd1-c9b8c074739b@redhat.com> Message-ID: <2a484676-cd50-77b1-c659-3b82bc7d511c@redhat.com> On 09/04/2019 19:30, Andrew Haley wrote: > On 4/9/19 11:42 AM, Andrew Dinn wrote: >> This new API method was conceived as a preliminary change for JEP 352 to >> allow selective writeback of NVRAM-backed buffers. However, it has been >> implemented to provide a similar capability for file-mapped byte >> buffers. The old brute-force API method, force(), continues to operate >> as before for file-mapped byte buffers. >> >> One detail that is worth highlighting is that for file-backed buffers >> the start address passed to the native method force0 is rounded down to >> a page boundary. This is needed for Unix implementations to ensure that >> the underlying msync system call does not throw an exception. > > Is it actually necessary to use a system call to do this? I would have > thought that NVRAM allowed a finer granularity than a whole page, too. It is necessary to use a system call to write back a normal, file-backed buffer, whether that is for the whole buffer (original force() method) or some subset of it (new force(int,int) method). That case is all that the current patch is concerned with. When NVRAM support is added the force operation will, of course, employ a cache line writeback and will only do so for the lines that overlap the region (i.e. it will operate at cache-line granularity rather than file page granularity). The prototype implementation of NVRAM support posted in earlier discussions shows how to do that, including how to ensure that the writeback is done using an inlined machine instruction. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Wed Apr 10 09:23:28 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 10 Apr 2019 10:23:28 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <32C928B7-C744-425F-8F9B-A02CD3AFBCE1@oracle.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> <32C928B7-C744-425F-8F9B-A02CD3AFBCE1@oracle.com> Message-ID: <562790ea-018c-862c-b528-f853ea921420@redhat.com> On 09/04/2019 19:15, Brian Burkhalter wrote: > >> On Apr 9, 2019, at 10:59 AM, Alan Bateman > > wrote: >> >> There are a couple of implementation details to discuss > > I was wondering whether this topic has any synergy / overlap with this > prior thread [1] the principal sticking point of which was IIRC forcing > / loading a slice of a MBB. > > http://mail.openjdk.java.net/pipermail/nio-dev/2019-February/005871.html Oh, bonus points for that question ;-) Currently, for file-backed MBBs, forcing a slice of the MBB has no effect. That happens because the slice is not provided with an fd when it is cloned off the MBB. The same applies when creating a view over the byte buffer memory using some other type e.g. as an int buffer. I don't know exactly why that decision was made (avoiding profileration of fds?) but anyway it is a fait accompli and it means you can only force writes to a slice or view using the original MBB. When creating a slice (or view) over an NVRAM-backed buffer the situation is different. It would be perfectly possible for the slice/view to record that it's originating MBB is backed by NVRAM without needing to store the fd in the slice/view. It does store the base address of the originating MBB and its own offset/scaling. So, it could easily translate the force indices in its own coordinate system into a corresponding region of the original byte buffer and then write back the relevant cache lines in the original MBB byte address range. Much as the above capability would be useful it would mean that NVRAM-backed buffer slices/views had different semantics to file-backed backed buffer slices/views. The prototype implementation I provided earlier chose to follow the existing behaviour so as not to open up any such difference. I am interested to hear if anyone thinks the alternative choice is worth pursuing. It would certainly make life easier for users of NVRAM-backed buffers as it would enable them to use the new flush API to flush regions using slice/view coordinates rather than translated coordinates for the original byte buffer. However, providing this capability only for NVRAM-backed buffers leaves the solution somewhat half-baked. No doubt, existing users of file-backed MBB slices/views have already worked round the current limitation by retaining a handle on the original MBB and flushing it at suitable points using a full force() call. However, those existing implementations will not have had to deal with coordinate translation since the flush is indiscriminate. If they wish to move to using this new region-based flush then they will have to record and translate buffer coordinates. I would prefer this requirement to be fully present or fully removed rather than imposing it for one case but not the other. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Wed Apr 10 09:44:10 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 10 Apr 2019 10:44:10 +0100 Subject: RFR: 8221397 Support implementation-defined Map Modes In-Reply-To: <98be0420-303e-3b4a-ed83-dc75996fa0a1@redhat.com> References: <96710df4-741a-99e0-1b5b-4e0dda538931@redhat.com> <5607e9c2-1f84-7ab5-42e9-4da1cb996dbe@redhat.com> <9946f451-db2d-5662-87ba-cbb527afc108@oracle.com> <296e270c-0db1-1cdb-c4c3-147cbfd417ea@redhat.com> <1de0a2d6-ab0a-4e02-9216-75a2e545bc87@oracle.com> <298ca076-64be-0a6c-cd77-ced66e5468c0@oracle.com> <27be3165-74f7-6d55-2d1e-57d5afd00c8d@redhat.com> <2026d6f7-3133-c6b9-ea50-18984d624810@oracle.com> <98be0420-303e-3b4a-ed83-dc75996fa0a1@redhat.com> Message-ID: On 10/04/2019 09:34, Andrew Dinn wrote: > : > Does that mean it is ok to push this now or do I need a second reviewer? > One Reviewer is fine but you need to finalize the CSR and wait for that to be approved before pushing. -Alan From adinn at redhat.com Wed Apr 10 11:15:19 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 10 Apr 2019 12:15:19 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> Message-ID: On 09/04/2019 18:59, Alan Bateman wrote: > There are a couple of implementation details to discuss but I'll stick > to the API/javadoc for now so that you can create the CSR. Ok, thanks. > We discussed the method signature, parameter types, and the semantics in > previous mails so I think they are all okay (meaning the first parameter > is the index in the buffer of the first byte to write back and not an > offset from the position). One nit is that the existing absolute methods > name the parameter "index" and I think we should do the same here. Yes, that is clearer as well as consistent. > In the first sentence it might be better to say "a region" rather "some > region". You could expand this paragraph with a second sentence, > something like "The region starts at the given {@code index} in this > buffer and is {@code length} bytes". Ok. > A suggestion for the description for the index and length parameters is > copy the wording/style from the existing absolute methods, e.g. > > @param index The index of the first byte of the regsion; must be > non-negative and less than limit() > @param length The length of the region in bytes; must be non-negative > and no larger than limit() - index > @throws IndexOutOfBoundException If the preconditions on the index and > length do not hold. Ok. An updated webrev is available: JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.01 I will also create a CSR based on the changes in this webrev and post an RFR. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From adinn at redhat.com Wed Apr 10 12:00:15 2019 From: adinn at redhat.com (Andrew Dinn) Date: Wed, 10 Apr 2019 13:00:15 +0100 Subject: RFR: CSR: JDK-8222261: MappedByteBuffer.force method to specify range Message-ID: Could I please get a review for the following CSR which accompanies JDK-8221696 JIRA: https://bugs.openjdk.java.net/browse/JDK-8222261 regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From Alan.Bateman at oracle.com Thu Apr 11 10:34:57 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 11 Apr 2019 11:34:57 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> Message-ID: On 10/04/2019 12:15, Andrew Dinn wrote: > : > An updated webrev is available: > > JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 > webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.01 > > The updated javadoc looks good. A minor nit is that it reads "and includes {@code length} bytes" when it should say "and is {@code length} bytes". There is also a stray quote at the end of the paragraph. I've added myself as Reviewer to the CSR so you can finalize it. -Alan From adinn at redhat.com Thu Apr 11 10:56:26 2019 From: adinn at redhat.com (Andrew Dinn) Date: Thu, 11 Apr 2019 11:56:26 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> Message-ID: <00ae2771-ba76-9ff1-5fbc-af487e0da3e7@redhat.com> On 11/04/2019 11:34, Alan Bateman wrote: > On 10/04/2019 12:15, Andrew Dinn wrote: >> : >> An updated webrev is available: >> >> JIRA:?? https://bugs.openjdk.java.net/browse/JDK-8221696 >> webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.01 >> >> > The updated javadoc looks good. A minor nit is that it reads "and > includes {@code length} bytes" when it should say "and is {@code length} > bytes". There is also a stray quote at the end of the paragraph. Ok, I have corrected that and will update the patch accordingly > I've added myself as Reviewer to the CSR so you can finalize it. Excellent, thank you. It has now been finalized. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From christoph.langer at sap.com Thu Apr 11 14:06:06 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Thu, 11 Apr 2019 14:06:06 +0000 Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 Message-ID: Hi, working on JDK-8213031 to add posix file permission support for the zipfs, I thought it makes sense to factor out some of the changes into a separate change. Those changes reorganize and clean up some parts of the code which I think makes sense in any case. Please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8222276 Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8222276.0/ In detail: The main change is to the hierarchy of inner classes ZipFileSystem.IndexNode and ZipFileSystem.Entry (the latter extends the former). Both classes are static classes currently. It makes sense, though, to have those associated with the ZipFileSystem they belong to. To do that, I need to add a static superclass named ZFSNode which only holds a node name and its hashcode. This class needs to exist because of the lookup implementation to find specific nodes of tze ZipFileSystem in its node map. Then the classes IndexNode extends ZFSNode and Entry extends IndexNode can be made non-static. Further changes are: Improve error handling for ZipFileAttributeView::get methods improve handling for zip64 attribute/version minor clenaups such as adding @Override annotations, whitespace fixes, private method visibility etc. Furthermore, there is some suspicious coding in JarFileSystem:: createVersionedLinks where a temporary key node is inserted into the alias map. However, this is a singleton instance which always gets new values. I'll open a separate bug for that and I'll try to create a test case which demonstrates an issue with the current code. I ran the zipfs jtreg tests successfully and will put it into our internal system as well as run it through jdk-submit. Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Thu Apr 11 21:36:26 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 11 Apr 2019 14:36:26 -0700 Subject: 8220477: Channels.newWriter() does not close if underlying channel throws an IOException In-Reply-To: <1A18A1EC-D7C8-4AA9-AFB8-5DB39166C6BC@oracle.com> References: <94010bf5-c219-5f67-f939-b8757e418adb@oracle.com> <1A18A1EC-D7C8-4AA9-AFB8-5DB39166C6BC@oracle.com> Message-ID: > On Mar 29, 2019, at 7:34 AM, Brian Burkhalter wrote: > >> On Mar 29, 2019, at 12:35 AM, Alan Bateman > wrote: >> >> On 28/03/2019 21:29, Brian Burkhalter wrote: >>> For the issue [1] please review the proposed patch [2]. The change is to catch an IOException thrown by implClose() and mark the {en,de}coder as closed anyway. Lots of lines of test for a small change! >>> >> Looks good. As you are there, would you mind fixing formatting in StreamDecoder? I can't tell what happened to it but it looks like the intendation has been removed from L230+ onwards (I only noticed it when looking at the implClose at the end of file). > > Yes I can fix that. > > After I posted this RFR, I ran the patch through all the testing tiers and with this patch the test > > sun/nio/cs/StreamEncoderClose.java > > fails at line 50. I need to think whether this indicates a problem with the patch or the StreamEncoderClose test is incorrect. Revisiting the failure mentioned above, it turns out that this is because a StreamEncoder for an OutputStream does not flush the stream before closing it. It seems like it should. (I don?t know whether this should perhaps be the subject of a separate issue.) The webrev [1] for the current issue 8220477 is updated to do this. The difference versus the previous webrev is [2]. It might be overkill to handle the IOException from OutputStream.flush() instead of just ignoring it, I?m not sure. Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8220477/webrev.01 [2] diff vs. webrev.00 --- a/src/java.base/share/classes/sun/nio/cs/StreamEncoder.java +++ b/src/java.base/share/classes/sun/nio/cs/StreamEncoder.java @@ -346,8 +346,26 @@ writeBytes(); if (ch != null) ch.close(); - else - out.close(); + else { + IOException flushException = null; + try { + out.flush(); + } catch (IOException fe) { + flushException = fe; + } finally { + try { + out.close(); + } catch (IOException closeException) { + if (flushException != null) { + closeException.addSuppressed(flushException); + } + throw closeException; + } + if (flushException!= null) { + throw flushException; + } + } + } } catch (IOException x) { encoder.reset(); throw x; -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Fri Apr 12 07:18:46 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 12 Apr 2019 08:18:46 +0100 Subject: 8220477: Channels.newWriter() does not close if underlying channel throws an IOException In-Reply-To: References: <94010bf5-c219-5f67-f939-b8757e418adb@oracle.com> <1A18A1EC-D7C8-4AA9-AFB8-5DB39166C6BC@oracle.com> Message-ID: <34946f22-b874-690f-a3c1-e5ca35833a37@oracle.com> On 11/04/2019 22:36, Brian Burkhalter wrote: > > Revisiting the failure mentioned above, it turns out that this is > because a StreamEncoder for an OutputStream does not flush the stream > before closing it. It seems like it should. (I don?t know whether this > should perhaps be the subject of a separate issue.) The webrev [1] for > the current issue?8220477 is updated to do this. The difference versus > the previous webrev is [2]. It might be overkill to handle the > IOException from OutputStream.flush() instead of just ignoring it, I?m > not sure. > It seems reasonable but it adds complexity to prefer the exception from flush over the close. The alternative is the much simpler: try { ??? out.flush(); } finally { ??? out.close(); } As always, if the flush fails then it's possible that the close will attempt to flush again so there may be several suppressed exceptions. -Alan From shade at redhat.com Fri Apr 12 13:34:09 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 12 Apr 2019 15:34:09 +0200 Subject: RFR (XS) 8222410: java/nio/file/attribute/BasicFileAttributeView/UnixSocketFile hangs when "nc" does not accept "-U" Message-ID: <14fc9eb0-4e3a-445a-b450-52788a9db72c@redhat.com> Test bug: https://bugs.openjdk.java.net/browse/JDK-8222410 Fix: http://cr.openjdk.java.net/~shade/8222410/webrev.01/ Testing: affected tests on systems that accept/reject -U, jdk-submit (running) -- Thanks, -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From Alan.Bateman at oracle.com Fri Apr 12 14:03:54 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 12 Apr 2019 15:03:54 +0100 Subject: RFR (XS) 8222410: java/nio/file/attribute/BasicFileAttributeView/UnixSocketFile hangs when "nc" does not accept "-U" In-Reply-To: <14fc9eb0-4e3a-445a-b450-52788a9db72c@redhat.com> References: <14fc9eb0-4e3a-445a-b450-52788a9db72c@redhat.com> Message-ID: <5e230fa2-ada2-8546-7c46-d31376d025dd@oracle.com> On 12/04/2019 14:34, Aleksey Shipilev wrote: > Test bug: > https://bugs.openjdk.java.net/browse/JDK-8222410 > > Fix: > http://cr.openjdk.java.net/~shade/8222410/webrev.01/ > > Testing: affected tests on systems that accept/reject -U, jdk-submit (running) > Can the `which nc` check? be removed as part of this? Also we should "Netcat" to "netcat" too (the inconsistency seems to be in the original test). -Alan From brian.burkhalter at oracle.com Fri Apr 12 14:16:17 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Fri, 12 Apr 2019 07:16:17 -0700 Subject: 8220477: Channels.newWriter() does not close if underlying channel throws an IOException In-Reply-To: <34946f22-b874-690f-a3c1-e5ca35833a37@oracle.com> References: <94010bf5-c219-5f67-f939-b8757e418adb@oracle.com> <1A18A1EC-D7C8-4AA9-AFB8-5DB39166C6BC@oracle.com> <34946f22-b874-690f-a3c1-e5ca35833a37@oracle.com> Message-ID: <0A893826-8F84-475B-ADF8-DE353E7057C1@oracle.com> > On Apr 12, 2019, at 12:18 AM, Alan Bateman wrote: > > On 11/04/2019 22:36, Brian Burkhalter wrote: >> >> Revisiting the failure mentioned above, it turns out that this is because a StreamEncoder for an OutputStream does not flush the stream before closing it. It seems like it should. (I don?t know whether this should perhaps be the subject of a separate issue.) The webrev [1] for the current issue 8220477 is updated to do this. The difference versus the previous webrev is [2]. It might be overkill to handle the IOException from OutputStream.flush() instead of just ignoring it, I?m not sure. >> > It seems reasonable but it adds complexity to prefer the exception from flush over the close. The alternative is the much simpler: > > try { > out.flush(); > } finally { > out.close(); > } > > As always, if the flush fails then it's possible that the close will attempt to flush again so there may be several suppressed exceptions. Indeed that is cleaner. Update webrev [1]. Diff versus original webrev [2]. I?ll not push this in any case until next week. Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8220477/webrev.02/ [2] diff vs. webrev.00 --- a/src/java.base/share/classes/sun/nio/cs/StreamEncoder.java +++ b/src/java.base/share/classes/sun/nio/cs/StreamEncoder.java @@ -346,8 +346,13 @@ writeBytes(); if (ch != null) ch.close(); - else - out.close(); + else { + try { + out.flush(); + } finally { + out.close(); + } + } } catch (IOException x) { encoder.reset(); throw x; -------------- next part -------------- An HTML attachment was scrubbed... URL: From shade at redhat.com Fri Apr 12 20:01:24 2019 From: shade at redhat.com (Aleksey Shipilev) Date: Fri, 12 Apr 2019 22:01:24 +0200 Subject: RFR (XS) 8222410: java/nio/file/attribute/BasicFileAttributeView/UnixSocketFile hangs when "nc" does not accept "-U" In-Reply-To: <5e230fa2-ada2-8546-7c46-d31376d025dd@oracle.com> References: <14fc9eb0-4e3a-445a-b450-52788a9db72c@redhat.com> <5e230fa2-ada2-8546-7c46-d31376d025dd@oracle.com> Message-ID: On 4/12/19 4:03 PM, Alan Bateman wrote: > On 12/04/2019 14:34, Aleksey Shipilev wrote: >> Test bug: >> ?? https://bugs.openjdk.java.net/browse/JDK-8222410 >> >> Fix: >> ?? http://cr.openjdk.java.net/~shade/8222410/webrev.01/ >> >> Testing: affected tests on systems that accept/reject -U, jdk-submit (running) >> > Can the `which nc` check? be removed as part of this? Maybe? I am leaning towards keeping both checks intact, so test logs clearly say what did happen. Also, I don't have a system without nc to test it properly. -Aleksey -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From Alan.Bateman at oracle.com Sun Apr 14 15:24:45 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Sun, 14 Apr 2019 16:24:45 +0100 Subject: RFR (XS) 8222410: java/nio/file/attribute/BasicFileAttributeView/UnixSocketFile hangs when "nc" does not accept "-U" In-Reply-To: References: <14fc9eb0-4e3a-445a-b450-52788a9db72c@redhat.com> <5e230fa2-ada2-8546-7c46-d31376d025dd@oracle.com> Message-ID: <79cc0e4e-75e9-798a-cc99-00f404d7cdd2@oracle.com> On 12/04/2019 21:01, Aleksey Shipilev wrote: > : > Maybe? I am leaning towards keeping both checks intact, so test logs clearly say what did happen. > Also, I don't have a system without nc to test it properly. > It does feel redundant but it's harmless so okay. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.burkhalter at oracle.com Tue Apr 16 17:59:54 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 16 Apr 2019 10:59:54 -0700 Subject: 8220477: Channels.newWriter() does not close if underlying channel throws an IOException In-Reply-To: <0A893826-8F84-475B-ADF8-DE353E7057C1@oracle.com> References: <94010bf5-c219-5f67-f939-b8757e418adb@oracle.com> <1A18A1EC-D7C8-4AA9-AFB8-5DB39166C6BC@oracle.com> <34946f22-b874-690f-a3c1-e5ca35833a37@oracle.com> <0A893826-8F84-475B-ADF8-DE353E7057C1@oracle.com> Message-ID: <49722C15-9961-4813-A8CD-95C2C3CAF893@oracle.com> > On Apr 12, 2019, at 7:16 AM, Brian Burkhalter wrote: > >> It seems reasonable but it adds complexity to prefer the exception from flush over the close. The alternative is the much simpler: >> >> try { >> out.flush(); >> } finally { >> out.close(); >> } >> >> As always, if the flush fails then it's possible that the close will attempt to flush again so there may be several suppressed exceptions. > > Indeed that is cleaner. Update webrev [1]. Diff versus original webrev [2]. > > I?ll not push this in any case until next week. This version passed the CI testing cleanly so is ready to go, I think. Brian > [1] http://cr.openjdk.java.net/~bpb/8220477/webrev.02/ > [2] diff vs. webrev.00 > > --- a/src/java.base/share/classes/sun/nio/cs/StreamEncoder.java > +++ b/src/java.base/share/classes/sun/nio/cs/StreamEncoder.java > @@ -346,8 +346,13 @@ > writeBytes(); > if (ch != null) > ch.close(); > - else > - out.close(); > + else { > + try { > + out.flush(); > + } finally { > + out.close(); > + } > + } > } catch (IOException x) { > encoder.reset(); > throw x; -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Wed Apr 17 07:10:20 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 17 Apr 2019 08:10:20 +0100 Subject: 8220477: Channels.newWriter() does not close if underlying channel throws an IOException In-Reply-To: <49722C15-9961-4813-A8CD-95C2C3CAF893@oracle.com> References: <94010bf5-c219-5f67-f939-b8757e418adb@oracle.com> <1A18A1EC-D7C8-4AA9-AFB8-5DB39166C6BC@oracle.com> <34946f22-b874-690f-a3c1-e5ca35833a37@oracle.com> <0A893826-8F84-475B-ADF8-DE353E7057C1@oracle.com> <49722C15-9961-4813-A8CD-95C2C3CAF893@oracle.com> Message-ID: <9de5e905-716b-f937-ca15-fdff463e378a@oracle.com> On 16/04/2019 18:59, Brian Burkhalter wrote: > : > > This version passed the CI testing cleanly so is ready to go, I think. > Great! From adinn at redhat.com Thu Apr 18 09:17:56 2019 From: adinn at redhat.com (Andrew Dinn) Date: Thu, 18 Apr 2019 10:17:56 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> Message-ID: <9218d37f-9820-9216-b125-8f3bc2d19e3d@redhat.com> Hi Alan, The CSR for this issue has been finalized and is awaiting approval. Would it now be possible to proceed with reviews of the implementation? JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.01 regards, Andrew Dinn ----------- From claes.redestad at oracle.com Thu Apr 18 13:10:32 2019 From: claes.redestad at oracle.com (Claes Redestad) Date: Thu, 18 Apr 2019 15:10:32 +0200 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel Message-ID: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> Hi, JDK-8034802 changed zipfs to defer deflation to only happen when closing/syncing the ZipFileSystem. Previously it happened eagerly when closing output streams retrieved via Files.newOutputStream. While this behavior change could have positive effects in some cases (since reading/writing entries don't need to inflate/deflate on the fly), it also broke the ability to deflate entries in parallel. This patch reverts the behavior so that entries are deflated (by default) when individual entry output streams are closed. File channels retain the behavior that entries are inflated into an internal byte array. Bug: https://bugs.openjdk.java.net/browse/JDK-8222532 Webrev: http://cr.openjdk.java.net/~redestad/8222532/open.00/ Testing: tier1-3, verified recuperation of performance in reproducer to pre-JDK-8034802 levels. Thanks! /Claes From brian.burkhalter at oracle.com Fri Apr 19 00:24:15 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Thu, 18 Apr 2019 17:24:15 -0700 Subject: 8220793: (fs) No support for changing modification time of symlink Message-ID: <94054704-7CD6-4FE8-BE86-D17145C06E97@oracle.com> For issue [1] please review the patch [2]. It implements setting the access and modification times of an unfollowed symbolic link on Linux and macOS. The patch uses the library function lutimes(3). The supported Unix platforms for lutimes() appear to be Linux and the BSDs, but neither Solaris (verified) nor AIX. Therefore the native function lutimes0() is a no-op for Unix platforms other than Linux and the BSDs. For purposes of testing, a version of the test without the @requires tag was used to enable its execution on Solaris. The results of running the test without and with the library changes applied are as follows: Test w/o fix Test with fix Linux Fail [3] Pass macOS Fail [3] Pass Solaris Fail [4] Fail [5] Windows Pass Pass For Solaris, this represents a behavioral change from throwing a FileSystemException to silently not changing the time attributes of the link. I don?t know whether this is an acceptable change. The possibility of falling back to utimes(2) in lutimes0() does not work as it modifies the time attributes of the file pointed to by the link but leaves those of the link itself unchanged. Thanks, Brian [1] https://bugs.openjdk.java.net/browse/JDK-8220793 [2] http://cr.openjdk.java.net/~bpb/8220793/webrev.00/ [3] FileSystemException from NIO java.nio.file.FileSystemException: /var/folders/m5/7p8b2jls02bb8w1q571kvtgh0000gq/T/name1606909272325238266/link: Too many levels of symbolic links or unable to access attributes of symbolic link at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:96) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.setTimes(UnixFileAttributeViews.java:85) at SymlinkTime.main(SymlinkTime.java:76) [4] FileSystemException from NIO java.nio.file.FileSystemException: /var/tmp/name18128007785949580669/link: Number of symbolic links encountered during path name traversal exceeds MAXSYMLINKS or unable to access attributes of symbolic link at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:96) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.setTimes(UnixFileAttributeViews.java:85) at SymlinkTime.main(SymlinkTime.java:76) [5] Test failure due to unexpected results java.lang.RuntimeException: Link - modification time: expected 2001-09-09T01:47:40Z, actual 2019-04-18T22:23:08.772611Z; access time: expected 2001-09-09T01:48:40Z, actual 2019-04-18T22:23:08.773325Z. at SymlinkTime.check(SymlinkTime.java:105) at SymlinkTime.main(SymlinkTime.java:83) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Apr 22 07:36:48 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 22 Apr 2019 08:36:48 +0100 Subject: 8220793: (fs) No support for changing modification time of symlink In-Reply-To: <94054704-7CD6-4FE8-BE86-D17145C06E97@oracle.com> References: <94054704-7CD6-4FE8-BE86-D17145C06E97@oracle.com> Message-ID: <6044b47b-74ff-1679-6b83-8cc1d71bcc39@oracle.com> On 19/04/2019 01:24, Brian Burkhalter wrote: > For issue [1] please review the patch [2]. It implements setting the > access and modification times of an unfollowed symbolic link on Linux > and macOS. > > The patch uses the library function lutimes(3). The supported Unix > platforms for lutimes() appear to be Linux and the BSDs, but neither > Solaris (verified) nor AIX. Therefore the native function lutimes0() > is a no-op for Unix platforms other than Linux and the BSDs. > I'm in two minds on this, mostly because it should be very rare to want to change the the time stamps of a symbolic link. In any case, can you double check setTimes - I assume it should not attempt to open the file when lutimes is available and the file is a sym link. In passing, I wonder if folks targeting embedded/older versions of glibc will complain. I see lutimes has been there since glibc 2.6. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Apr 22 11:08:27 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 22 Apr 2019 12:08:27 +0100 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks Message-ID: In JDK 11 we changed most of the locking in the network channels to use explicit locks in preparation for a future world of user-mode threads [1]. I'd like to do another round of update in this area to change the stateLock to an explicit lock and also re-visit the implementation of the socket adaptors that currently make use of the channel blockingLock. The motivation for changing stateLock is that closing a channel is a blocking operation when it must coordinate the close with threads that are blocked in I/O operations on the channel. Changing this to an explicit lock it a simple translation although it touches a lot of places. As part of this the closeLock used in AbstractInterruptibleChannel also needs to be changed. The changes to support the socket adaptors is a bit trickery. Timed connect/accept/read (via the adaptors) require the underlying socket to be temporarily changed to non-blocking mode whilst the channel remains in blocking mode. In the case of timed read then this should be possible while another thread is doing a blocking write - that blocking write will happen while the underlying socket is in non-blocking mode. To keep it simple, and avoid another round of changes later, all the blocking operations are changed so that they can work even if the underlying socket is non-blocking. While re-visiting the socket adaptors I spotted several issues/bugs that have not been reported. I'll create issues to follow-up on those. However in passing, there are a few small changes to the socket adaptor, one of which is to create each socket adaptos with a dummy SocketImpl. Michael McMahon is working on JDK-8216978 [2] to drop support for pre JDK 1.4 SocketImpls - if that change gets rid the annoying references from the SocketImpl to the enclosing Socket/ServerSocket then it will allow all socket adaptors to share a dummy SocketImpl. The existing tests provide reasonable coverage of the changes. The only area that wasn't tested (as it never really worked) is concurrent use of the input and output streams obtained from a socket adaptor. The new test attempts to exercise all the possible scenarios. There are two items that aren't addressed in this webrev: 1. DatagramSocket's socket adaptor. The send/receive methods defined by legacy DatagramSocket synchronizes on the datagram packet. It requires further investigation although I suspect it may not be feasible to do anything here for compatibility reasons. 2. Interrupting a thread blocked in I/O operation will close the channel so that makes Thread.interrupt a blocking operation. The changes for this to play well with user mode threads touches other areas so I'll separate that out for now. The webrev with the proposed changes is here: ?? http://cr.openjdk.java.net/~alanb/8222774/0/webrev/index.html -Alan [1] https://mail.openjdk.java.net/pipermail/nio-dev/2018-February/004683.html [2] https://bugs.openjdk.java.net/browse/JDK-8216978 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Mon Apr 22 15:52:14 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 22 Apr 2019 16:52:14 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <9218d37f-9820-9216-b125-8f3bc2d19e3d@redhat.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> <9218d37f-9820-9216-b125-8f3bc2d19e3d@redhat.com> Message-ID: <453045f6-0259-e098-8211-0187428adb9f@oracle.com> On 18/04/2019 10:17, Andrew Dinn wrote: > Hi Alan, > > The CSR for this issue has been finalized and is awaiting approval. > Would it now be possible to proceed with reviews of the implementation? > > JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 > webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.01 > The calculation to compute the page aligned address and then the distance from there to the end of the region is tricky to get right. I think you have it right, I'm just wondering if we could leverage the existing mappingOffset/mappingAddress methods so that we have only one place to audit. For example, suppose mappingOffset is changed to take an index. I think this would reduce the new force method down to: long offset = mappingOffset(index); force0(fd, mappingAddress(offset), offset + length); We'd need to add an overload so that the existing usages of mappingOffset would continue to work, or we change then to use mappingOffset(0).? Just something to think about because this code is tricky. I don't expect any issues on Windows but I will test it to make sure. -Alan From brian.burkhalter at oracle.com Mon Apr 22 18:19:00 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Mon, 22 Apr 2019 11:19:00 -0700 Subject: 8220793: (fs) No support for changing modification time of symlink In-Reply-To: <6044b47b-74ff-1679-6b83-8cc1d71bcc39@oracle.com> References: <94054704-7CD6-4FE8-BE86-D17145C06E97@oracle.com> <6044b47b-74ff-1679-6b83-8cc1d71bcc39@oracle.com> Message-ID: > On Apr 22, 2019, at 12:36 AM, Alan Bateman wrote: > > On 19/04/2019 01:24, Brian Burkhalter wrote: >> For issue [1] please review the patch [2]. It implements setting the access and modification times of an unfollowed symbolic link on Linux and macOS. >> >> The patch uses the library function lutimes(3). The supported Unix platforms for lutimes() appear to be Linux and the BSDs, but neither Solaris (verified) nor AIX. Therefore the native function lutimes0() is a no-op for Unix platforms other than Linux and the BSDs. >> > I'm in two minds on this, mostly because it should be very rare to want to change the the time stamps of a symbolic link. > > In any case, can you double check setTimes - I assume it should not attempt to open the file when lutimes is available and the file is a sym link. It will only if just one of the times is being modified (lines 99-111). Also I neglected to put it in the retry section (lines 136-148) and that should likely be added. > In passing, I wonder if folks targeting embedded/older versions of glibc will complain. I see lutimes has been there since glibc 2.6. glibc 2.6 was announced in May, 2007 it appears. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Mon Apr 22 19:26:33 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Mon, 22 Apr 2019 15:26:33 -0400 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> Message-ID: <833DD3A6-5E0C-4D2C-8C7E-E3BD305E5B63@oracle.com> Hi Claes, Thank you for your efforts here. I have gone through your changes, comparing to the original code prior to JDK-8034802 and your changes look good. It would be good if someone else can also help sanity check but you did a nice job of adding the functionality back. Before you push, please update the copyright for ZipFileSystemProvider. Best, Lance > On Apr 18, 2019, at 9:10 AM, Claes Redestad wrote: > > Hi, > > JDK-8034802 changed zipfs to defer deflation to only happen when > closing/syncing the ZipFileSystem. Previously it happened eagerly when > closing output streams retrieved via Files.newOutputStream. While this > behavior change could have positive effects in some cases (since > reading/writing entries don't need to inflate/deflate on the fly), it > also broke the ability to deflate entries in parallel. > > This patch reverts the behavior so that entries are deflated (by > default) when individual entry output streams are closed. File channels > retain the behavior that entries are inflated into an internal byte > array. > > Bug: https://bugs.openjdk.java.net/browse/JDK-8222532 > Webrev: http://cr.openjdk.java.net/~redestad/8222532/open.00/ > > Testing: tier1-3, verified recuperation of performance in reproducer to > pre-JDK-8034802 levels. > > Thanks! > > /Claes Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From claes.redestad at oracle.com Mon Apr 22 20:16:15 2019 From: claes.redestad at oracle.com (Claes Redestad) Date: Mon, 22 Apr 2019 22:16:15 +0200 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: <833DD3A6-5E0C-4D2C-8C7E-E3BD305E5B63@oracle.com> References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> <833DD3A6-5E0C-4D2C-8C7E-E3BD305E5B63@oracle.com> Message-ID: <5a1c485c-6d30-2557-675b-2321e81171a3@oracle.com> Hi Lance, On 2019-04-22 21:26, Lance Andersen wrote: > Hi Claes, > > Thank you for your efforts here. > > I have gone through your changes, comparing to the original code prior > to JDK-8034802 and your changes look good. ?It would be good if someone > else can ?also help sanity check but you did a nice job of adding the > functionality back. thanks for reviewing! And yes, I'd appreciate more reviews, too. :-) > > Before you push, please update the copyright for ZipFileSystemProvider. Sure Thanks! /Claes > > Best, > Lance >> On Apr 18, 2019, at 9:10 AM, Claes Redestad > > wrote: >> >> Hi, >> >> JDK-8034802 changed zipfs to defer deflation to only happen when >> closing/syncing the ZipFileSystem. Previously it happened eagerly when >> closing output streams retrieved via Files.newOutputStream. While this >> behavior change could have positive effects in some cases (since >> reading/writing entries don't need to inflate/deflate on the fly), it >> also broke the ability to deflate entries in parallel. >> >> This patch reverts the behavior so that entries are deflated (by >> default) when individual entry output streams are closed. File channels >> retain the behavior that entries are inflated into an internal byte >> array. >> >> Bug: https://bugs.openjdk.java.net/browse/JDK-8222532 >> Webrev: http://cr.openjdk.java.net/~redestad/8222532/open.00/ >> >> Testing: tier1-3, verified recuperation of performance in reproducer to >> pre-JDK-8034802 levels. >> >> Thanks! >> >> /Claes > > > > Lance Andersen| > Principal Member of Technical Staff | +1.781.442.2037 > Oracle?Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > From christoph.langer at sap.com Tue Apr 23 09:56:32 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Tue, 23 Apr 2019 09:56:32 +0000 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: <5a1c485c-6d30-2557-675b-2321e81171a3@oracle.com> References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> <833DD3A6-5E0C-4D2C-8C7E-E3BD305E5B63@oracle.com> <5a1c485c-6d30-2557-675b-2321e81171a3@oracle.com> Message-ID: Hi Claes, I checked the change and it looks fine to me, too. It contains some nice cleanups. I have a few nits, no need for a new webrev though: ZipFileSystem.java: - The added imports java.nio.channels.Channels and java.nio.channels.NonWritableChannelException don't seem to be necessary to me. - EntryOutputChannel::close (line 627 ff): Maybe you want to use try-with-resources for the output stream from getOutputStream(e) ? - line 634, private int getCompressMethod(FileAttribute... attrs) {... As you're about to clean it up, I guess you should remove the unused attrs from the method signature, too. - line 2510: 2510 private static class ExChannelCloser { -> there is a space too much between "ExChannelCloser" and "{" Best regards Christoph > -----Original Message----- > From: nio-dev On Behalf Of Claes > Redestad > Sent: Montag, 22. April 2019 22:16 > To: Lance Andersen > Cc: nio-dev at openjdk.java.net > Subject: Re: RFR: 8222532: (zipfs) Performance regression when writing > ZipFileSystem entries in parallel > > Hi Lance, > > On 2019-04-22 21:26, Lance Andersen wrote: > > Hi Claes, > > > > Thank you for your efforts here. > > > > I have gone through your changes, comparing to the original code prior > > to JDK-8034802 and your changes look good. ?It would be good if someone > > else can ?also help sanity check but you did a nice job of adding the > > functionality back. > > thanks for reviewing! And yes, I'd appreciate more reviews, too. :-) > > > > > Before you push, please update the copyright for ZipFileSystemProvider. > > Sure > > Thanks! > > /Claes > > > > Best, > > Lance > >> On Apr 18, 2019, at 9:10 AM, Claes Redestad >> > wrote: > >> > >> Hi, > >> > >> JDK-8034802 changed zipfs to defer deflation to only happen when > >> closing/syncing the ZipFileSystem. Previously it happened eagerly when > >> closing output streams retrieved via Files.newOutputStream. While this > >> behavior change could have positive effects in some cases (since > >> reading/writing entries don't need to inflate/deflate on the fly), it > >> also broke the ability to deflate entries in parallel. > >> > >> This patch reverts the behavior so that entries are deflated (by > >> default) when individual entry output streams are closed. File channels > >> retain the behavior that entries are inflated into an internal byte > >> array. > >> > >> Bug: https://bugs.openjdk.java.net/browse/JDK-8222532 > >> Webrev: http://cr.openjdk.java.net/~redestad/8222532/open.00/ > >> > >> Testing: tier1-3, verified recuperation of performance in reproducer to > >> pre-JDK-8034802 levels. > >> > >> Thanks! > >> > >> /Claes > > > > > > 198324.gif> > > Lance > Andersen| > > Principal Member of Technical Staff | +1.781.442.2037 > > Oracle?Java Engineering > > 1 Network Drive > > Burlington, MA 01803 > > Lance.Andersen at oracle.com > > > > > > From Alan.Bateman at oracle.com Tue Apr 23 10:42:32 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 23 Apr 2019 11:42:32 +0100 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> Message-ID: <72cb59c0-e977-d297-589f-80f7d1a417e4@oracle.com> On 18/04/2019 14:10, Claes Redestad wrote: > Hi, > > JDK-8034802 changed zipfs to defer deflation to only happen when > closing/syncing the ZipFileSystem. Previously it happened eagerly when > closing output streams retrieved via Files.newOutputStream. While this > behavior change could have positive effects in some cases (since > reading/writing entries don't need to inflate/deflate on the fly), it > also broke the ability to deflate entries in parallel. > > This patch reverts the behavior so that entries are deflated (by > default) when individual entry output streams are closed. File channels > retain the behavior that entries are inflated into an internal byte > array. > > Bug:??? https://bugs.openjdk.java.net/browse/JDK-8222532 > Webrev: http://cr.openjdk.java.net/~redestad/8222532/open.00/ I think the approach looks good. One thing that I think could be improved is ExChannelCloser. I think it needs a better name. Also I think it should encapsulate its fields a lot better so that sync doesn't need to access its interanls -- e.g. maybe it can define a closeAndDelete method that closes the channel and deletes the path. -Alan From claes.redestad at oracle.com Tue Apr 23 11:42:32 2019 From: claes.redestad at oracle.com (Claes Redestad) Date: Tue, 23 Apr 2019 13:42:32 +0200 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: <72cb59c0-e977-d297-589f-80f7d1a417e4@oracle.com> References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> <72cb59c0-e977-d297-589f-80f7d1a417e4@oracle.com> Message-ID: <1f20a70c-3d35-5005-12d4-d5a6853e4549@oracle.com> Hi Alan, Christoph, thanks for reviewing this! On 2019-04-23 12:42, Alan Bateman wrote: > On 18/04/2019 14:10, Claes Redestad wrote: >> Webrev: http://cr.openjdk.java.net/~redestad/8222532/open.00/ > I think the approach looks good. > > One thing that I think could be improved is ExChannelCloser. I think it > needs a better name. Also I think it should encapsulate its fields a lot > better so that sync doesn't need to access its interanls -- e.g. maybe > it can define a closeAndDelete method that closes the channel and > deletes the path. > Based on what it does, ExChannelCloser (which I restored from its pre- existing state) should probably be named ExistingChannelCloser. I've collected the feedback from you, Christoph and Lance into this new webrev: http://cr.openjdk.java.net/~redestad/8222532/open.01/ Incremental: http://cr.openjdk.java.net/~redestad/8222532/open.00_01/ Thanks! /Claes From daniel.fuchs at oracle.com Tue Apr 23 14:34:35 2019 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 23 Apr 2019 15:34:35 +0100 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks In-Reply-To: References: Message-ID: <6e17e946-aeae-f18b-f913-38f5f797d357@oracle.com> Hi Alan, I have read through the patch and didn't see anything too surprising. I might not have followed all the subtleties though - so another pair of eyes would be good. in DatagramChannelImpl.java, a few places use: 458 if (blocking) { 459 while (IOStatus.okayToRetry(n) && isOpen()) { ... } and a few others use 549 if (blocking && IOStatus.okayToRetry(n)) { 550 do { ... 553 } while (IOStatus.okayToRetry(n) && isOpen()); I was wondering about the asymmetry. It looks like the former if (blocking) { while() { ... } } might be more correct, as it ensure that isOpen() is also checked between the first and second call to the IO primitive which is being wrapped? (same goes for ServerSocketChannelImpl.java SocketChannelImpl.java, SinkChannelImpl.java) DatagramSocketAdaptor.java: 279 @Override 280 public int getSoTimeout() throws SocketException { 281 if (!dc.isOpen()) 282 throw new SocketException("Socket is closed"); Is there any backward compatibility concern with this change? I mean - I understand that setting a value on a closed socket makes no sense. But could some code out there be trying to figure out what was set on the socket before it was closed, e.g. for diagnosis purposes? (same goes for ServerSocketAdaptor.java) AdaptorStreams.java: Has this test run through tier8? This is where the special VM configurations happen - and it's been known to trigger intermittent failure more often than the other tiers. I see the test use 1000 and 2000 millis, reasoning that if the timeout is 2000ms then something scheduled to happen after 1000ms should not trigger the timeout - and I wonder how stable that is going to be. I see the test uses ServerSocket(0). We manage to eliminate most of the intermittent failures in the new HttpClient tests by consistently binding out server sockets to the loopback address, and avoiding the wildcard. I wonder if we should do it here too. best regards, -- daniel On 22/04/2019 12:08, Alan Bateman wrote: > > In JDK 11 we changed most of the locking in the network channels to use > explicit locks in preparation for a future world of user-mode threads > [1]. I'd like to do another round of update in this area to change the > stateLock to an explicit lock and also re-visit the implementation of > the socket adaptors that currently make use of the channel blockingLock. > > The motivation for changing stateLock is that closing a channel is a > blocking operation when it must coordinate the close with threads that > are blocked in I/O operations on the channel. Changing this to an > explicit lock it a simple translation although it touches a lot of > places. As part of this the closeLock used in > AbstractInterruptibleChannel also needs to be changed. > > The changes to support the socket adaptors is a bit trickery. Timed > connect/accept/read (via the adaptors) require the underlying socket to > be temporarily changed to non-blocking mode whilst the channel remains > in blocking mode. In the case of timed read then this should be possible > while another thread is doing a blocking write - that blocking write > will happen while the underlying socket is in non-blocking mode. To keep > it simple, and avoid another round of changes later, all the blocking > operations are changed so that they can work even if the underlying > socket is non-blocking. > > While re-visiting the socket adaptors I spotted several issues/bugs that > have not been reported. I'll create issues to follow-up on those. > However in passing, there are a few small changes to the socket adaptor, > one of which is to create each socket adaptos with a dummy SocketImpl. > Michael McMahon is working on JDK-8216978 [2] to drop support for pre > JDK 1.4 SocketImpls - if that change gets rid the annoying references > from the SocketImpl to the enclosing Socket/ServerSocket then it will > allow all socket adaptors to share a dummy SocketImpl. > > The existing tests provide reasonable coverage of the changes. The only > area that wasn't tested (as it never really worked) is concurrent use of > the input and output streams obtained from a socket adaptor. The new > test attempts to exercise all the possible scenarios. > > There are two items that aren't addressed in this webrev: > > 1. DatagramSocket's socket adaptor. The send/receive methods defined by > legacy DatagramSocket synchronizes on the datagram packet. It requires > further investigation although I suspect it may not be feasible to do > anything here for compatibility reasons. > > 2. Interrupting a thread blocked in I/O operation will close the channel > so that makes Thread.interrupt a blocking operation. The changes for > this to play well with user mode threads touches other areas so I'll > separate that out for now. > > The webrev with the proposed changes is here: > http://cr.openjdk.java.net/~alanb/8222774/0/webrev/index.html > > -Alan > > [1] > https://mail.openjdk.java.net/pipermail/nio-dev/2018-February/004683.html > [2] https://bugs.openjdk.java.net/browse/JDK-8216978 From Alan.Bateman at oracle.com Tue Apr 23 15:12:03 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 23 Apr 2019 16:12:03 +0100 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks In-Reply-To: <6e17e946-aeae-f18b-f913-38f5f797d357@oracle.com> References: <6e17e946-aeae-f18b-f913-38f5f797d357@oracle.com> Message-ID: <26704227-7873-3750-45a4-f9814327a266@oracle.com> On 23/04/2019 15:34, Daniel Fuchs wrote: > : > > I was wondering about the asymmetry. It looks like > the former if (blocking) { while() { ... } } might be more > correct, as it ensure that isOpen() is also checked between > the first and second call to the IO primitive which is > being wrapped? (same goes for ServerSocketChannelImpl.java > SocketChannelImpl.java, SinkChannelImpl.java) DatagramChannel.receive is a bit different as the non-blocking case may return null instead of a sender address.? However, point taken that we could change this to do-while so at least the blocking case is closer to the idiom used in the other locations. > > DatagramSocketAdaptor.java: > > ?279???? @Override > ?280???? public int getSoTimeout() throws SocketException { > ?281???????? if (!dc.isOpen()) > ?282???????????? throw new SocketException("Socket is closed"); > > Is there any backward compatibility concern with this > change? DatagramSocket.setSoTimeout has always thrown IAE for this case. Chris has picked up JDK-8222829 to align the spec with the long standing behavior. We had the same issue very recently with ServerSocket.setSoTimeout. > > AdaptorStreams.java: > > Has this test run through tier8? This is where the special > VM configurations happen - and it's been known to trigger > intermittent failure more often than the other tiers. > > I see the test use 1000 and 2000 millis, reasoning that if > the timeout is 2000ms then something scheduled to happen > after 1000ms should not trigger the timeout - and I wonder > how stable that is going to be. Daniel and I went through each of the tests using a timeout and I think we've satisfied ourselves that the timeouts are long enough so that there shouldn't be an issue on really slow test machines/ -Alan. From adinn at redhat.com Tue Apr 23 16:15:01 2019 From: adinn at redhat.com (Andrew Dinn) Date: Tue, 23 Apr 2019 17:15:01 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <453045f6-0259-e098-8211-0187428adb9f@oracle.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> <9218d37f-9820-9216-b125-8f3bc2d19e3d@redhat.com> <453045f6-0259-e098-8211-0187428adb9f@oracle.com> Message-ID: Hi Alan, Thanks for looking at this. On 22/04/2019 16:52, Alan Bateman wrote: > The calculation to compute the page aligned address and then the > distance from there to the end of the region is tricky to get right. I > think you have it right, I'm just wondering if we could leverage the > existing mappingOffset/mappingAddress methods so that we have only one > place to audit. For example, suppose mappingOffset is changed to take an > index. I think this would reduce the new force method down to: > > long offset = mappingOffset(index); > force0(fd, mappingAddress(offset), offset + length); I like this idea but I'd like to make sure I am clear about exactly how it ought to work. If you will indulge me I will work through this by describing first the status quo (as I perceive it) and then what my change attempts before trying to reconcile the two. Currently: There are 4 key locations of interest in the mapped byte range [... map_base ... address ... address+limit ... address+capacity ...] | | +----- page size -----+ where: - addresses to the left are numerically (i.e. as longs) lower than those to the right - map_base is aligned to page size -- it is obtained by rounding down address to a page boundary - consequently (address - map_base) is less than page size n.b. the mmapped region may or may not commence at map_base but it will always include map_base. Also, it will include bytes at all subsequent addresses up to (address + capacity). Whether it includes or extends beyond (address + capacity) is not really important. mappingOffset() computes (address - map_base): int ps = Bits.pageSize(); long offset = address % ps; return (offset >= 0) ? offset : (ps + offset); The jiggery-pokery with ? : works round the rather unhelpful behaviour of mod wrt negative longs. I think it could, equivalently, compute it directly as: int ps = Bits.pageSize(); long map_base = (address & ~(ps - 1)); return address - map_base; mappingAddress(mappingOffset) converts a previously retrieved mappingOffset back to the map base: return (address - mappingOffset) == (address - (address - map_base)) == map_base mappingLength(mappingOffset) computes ((address + capacity) - map_base) i.e. the byte count from start of page map_base to the last (potentially) /writeable/ buffer byte. return capacity() + mappingOffset == capacity + (address - map_base) == (address + capacity) - map_base arguably this method could just compute ((address + length) - map_base) i.e. the byte count from map_base to the last /written/ byte -- but that makes no real difference when performing a file-based flush. The call to force0 is made as force0(fd, mappingAddress(offset), mappingLength(offset)) which is equivalent to force0(fd, map_base, capacity + (address - map_base)) My updated code: When looking at an indexed location there are now 7 key locations of interest in the mapped byte range [... m_base ... a ... i_base ... a+i ... a+i+ln ... a+lim, a+cap ...] | | | | +- page size -+ +- page size -+ where: - for brevity, map_base is abbreviated to m_base, address to a, index_base to i_base, index to i, length to ln, limit to lim and capacity to cap - index is identifies the start of the region to be flushed - length is the length of the region to be flushed - (address + index) is the address of the byte at offset index - index_base is the largest page aligned address below (address + idx) i.e. it is obtained by rounding down the first byte of the flush region to a page boundary index_base is computed as follows int ps = Bits.pageSize(); long index_base = ((address + index) & ~(ps - 1)); My code is calling force0(fd, index_base, length + ((address + index) - i_base)) i.e. it the flushed range starts at the nearest page boundary below or equal to (address + index) and includes the original length bytes plus the intervening bytes between that page boundary and address. When this is considered by analogy with the previous call then three things stand out: 1) in place of map_base for the first argument we have index_base 2) in place of the extra included intervening bytes between map_base and address we have the extra included intervening bytes between index_base and address + index 3) in place of capacity (default) demarcated bytes to be flushed we have length user demarcated bytes to be flushed So, to retain those parallels detailed in 1, 2 and 3 we need mappingOffset and mappingAddress to be overloaded and the originals rerouted as follows mappingOffset(index) { int ps = Bits.pageSize(); long index_address = address + index; long index_base = (index_address & ~(ps - 1)); return index_address - index_base; } mappingOffset() => mappingOffset(0) mappingAddress(mappingOffset, index) { long index_address = address + index; return index_address - mappingOffset == (index_address - (index_address - index_base)) == index_base } mappingAddress(mappingOffset) => mappingAddress(mappingOffset, 0) mappingLength(mappingOffset, length) { return length + mappingOffset } mappingLength(mappingOffset) => mappingLength(mappingOffset, (long)capacity()) Does that look correct? If so I will prepare a new webrev accordingly. > I don't expect any issues on Windows but I will test it to make sure. Thank you. That would be very helpful. regards, Andrew Dinn ----------- Senior Principal Software Engineer Red Hat UK Ltd Registered in England and Wales under Company Registration No. 03798903 Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander From lance.andersen at oracle.com Tue Apr 23 17:23:50 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Tue, 23 Apr 2019 13:23:50 -0400 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: <1f20a70c-3d35-5005-12d4-d5a6853e4549@oracle.com> References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> <72cb59c0-e977-d297-589f-80f7d1a417e4@oracle.com> <1f20a70c-3d35-5005-12d4-d5a6853e4549@oracle.com> Message-ID: Hi Claes, The additional updates look fine. :-) Best Lance > On Apr 23, 2019, at 7:42 AM, Claes Redestad wrote: > > Hi Alan, Christoph, > > thanks for reviewing this! > > On 2019-04-23 12:42, Alan Bateman wrote: >> On 18/04/2019 14:10, Claes Redestad wrote: >>> Webrev: http://cr.openjdk.java.net/~redestad/8222532/open.00/ >> I think the approach looks good. >> One thing that I think could be improved is ExChannelCloser. I think it needs a better name. Also I think it should encapsulate its fields a lot better so that sync doesn't need to access its interanls -- e.g. maybe it can define a closeAndDelete method that closes the channel and deletes the path. > > Based on what it does, ExChannelCloser (which I restored from its pre- > existing state) should probably be named ExistingChannelCloser. > > I've collected the feedback from you, Christoph and Lance into this > new webrev: http://cr.openjdk.java.net/~redestad/8222532/open.01/ > > Incremental: http://cr.openjdk.java.net/~redestad/8222532/open.00_01/ > > Thanks! > > /Claes Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From brian.burkhalter at oracle.com Tue Apr 23 18:30:11 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 23 Apr 2019 11:30:11 -0700 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks In-Reply-To: <6e17e946-aeae-f18b-f913-38f5f797d357@oracle.com> References: <6e17e946-aeae-f18b-f913-38f5f797d357@oracle.com> Message-ID: <67F734F5-A7EE-4C01-8555-468176396CD5@oracle.com> Hi Alan, Daniel, > On Apr 23, 2019, at 7:34 AM, Daniel Fuchs wrote: > > I have read through the patch and didn't see anything > too surprising. I might not have followed all the subtleties > though - so another pair of eyes would be good. I went through the patch yesterday and this morning and do not see any problems either. Missing javadoc: src/java.base/share/classes/sun/nio/ch/SelChImpl.java Line 95: missing documentation before park(int) > On Apr 23, 2019, at 8:12 AM, Alan Bateman wrote: > >> AdaptorStreams.java: >> >> Has this test run through tier8? This is where the special >> VM configurations happen - and it's been known to trigger >> intermittent failure more often than the other tiers. >> >> I see the test use 1000 and 2000 millis, reasoning that if >> the timeout is 2000ms then something scheduled to happen >> after 1000ms should not trigger the timeout - and I wonder >> how stable that is going to be. > Daniel and I went through each of the tests using a timeout and I think we've satisfied ourselves that the timeouts are long enough so that there shouldn't be an issue on really slow test machines/ When the test is run on my machine without the source patch applied it timed out in testConcurrentReadWrite1() but passes with the patch. The @bug tag lists 4430139 which I suppose is intentional. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From martinrb at google.com Tue Apr 23 19:32:36 2019 From: martinrb at google.com (Martin Buchholz) Date: Tue, 23 Apr 2019 12:32:36 -0700 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks In-Reply-To: References: Message-ID: MILLISECONDS.convert(nanos, NANOSECONDS); At Google we prefer the explicit methods (here toMillis) for clarity, when possible. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martinrb at google.com Tue Apr 23 19:59:48 2019 From: martinrb at google.com (Martin Buchholz) Date: Tue, 23 Apr 2019 12:59:48 -0700 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks In-Reply-To: References: Message-ID: default void park(int event, I would not have named a wait-for-network operation "park" - it's not really related to LockSupport.park; there's no unpark operation. The phrase "unless the permit is available" looks like it was copied from the park spec - does it really make sense here? Is "event" really a single event, or an "event set"? --- "explicit locks" Can you refer to these as j.u.c. locks (or perhaps simply ReentrantLock) since even builtin locks are "explicit"? --- I'm a little surprised at the pervasive conversion from builtin locks. For loom to succeed, you'll have to make builtin locks work eventually, eh? History: back in 1.5, j.u.c. avoided builtin locks because ReentrantLock used to be more efficient, and for dogfooding. But it's hard to compete with hotspot and so today the (rather weak) recommendation for users is to use builtin locks unless the added flexibility or clarity of j.u.c. locks is needed. (of course, the deep support for builtin locks in hotspot is also the root cause of loom conversion trouble) --- (sorry, still not a network engineer) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Apr 23 20:05:23 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 23 Apr 2019 21:05:23 +0100 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks In-Reply-To: <67F734F5-A7EE-4C01-8555-468176396CD5@oracle.com> References: <6e17e946-aeae-f18b-f913-38f5f797d357@oracle.com> <67F734F5-A7EE-4C01-8555-468176396CD5@oracle.com> Message-ID: <7a262de6-db81-e347-c0aa-964ea3f2b422@oracle.com> On 23/04/2019 19:30, Brian Burkhalter wrote: > When the test is run on my machine without the source patch applied it > timed out in testConcurrentReadWrite1() but passes with the patch. > > The @bug tag lists 4430139 which I suppose is intentional. > Right, all of testConcurrentXXX tests will hang with jdk/jdk as the adaptors have never supported concurrent operations. Yes, the @bug line is intensional as the new test covers the JDK-4430139 (and a lot more) so the specific old test is removed. Daniel sits nearby and we talked a bit about the loops to deal with EINTR/retry. They are changed to while loops to make them a bit easier to reader. I see Martin's comment about the 4 cases where timeouts are using the 2-arg convert rather than toMillis or toNanos so I've changed those too. The update webrev is here: http://cr.openjdk.java.net/~alanb/8222774/1/webrev/index.html -Alan From Alan.Bateman at oracle.com Tue Apr 23 20:23:06 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 23 Apr 2019 21:23:06 +0100 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks In-Reply-To: References: Message-ID: <0b256e3f-c699-5843-0fb3-011a5a3751b9@oracle.com> On 23/04/2019 20:59, Martin Buchholz wrote: > default void park(int event, > > I would not have named a wait-for-network operation "park" - it's not > really related to LockSupport.park; there's no unpark operation.? The > phrase "unless the permit is available" looks like it was copied from > the park spec - does it really make sense here?? ?Is "event" really a > single event, or an "event set"? Once all the pieces are in place then it will work like park/unpark. I didn't mean to bring over the complete comment so I'll fix that. Blocking operations on channels wait for a single event. It would be possible to poll for multiple events but there isn't anything in the API that supports that (this is all for blocking operations so there are no Selectors in the picture). > : > > I'm a little surprised at the pervasive conversion from builtin > locks.? For loom to succeed, you'll have to make builtin locks work > eventually, eh? > I expect so but maybe not initially. -Alan From brian.burkhalter at oracle.com Tue Apr 23 21:12:33 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 23 Apr 2019 14:12:33 -0700 Subject: 8222774: (ch) Replace uses of stateLock and blockingLock with explicit locks In-Reply-To: <7a262de6-db81-e347-c0aa-964ea3f2b422@oracle.com> References: <6e17e946-aeae-f18b-f913-38f5f797d357@oracle.com> <67F734F5-A7EE-4C01-8555-468176396CD5@oracle.com> <7a262de6-db81-e347-c0aa-964ea3f2b422@oracle.com> Message-ID: <4DCF5A7D-4CAE-4725-9253-1CF8ABB058E7@oracle.com> > On Apr 23, 2019, at 1:05 PM, Alan Bateman wrote: > > Daniel sits nearby and we talked a bit about the loops to deal with EINTR/retry. They are changed to while loops to make them a bit easier to reader. I see Martin's comment about the 4 cases where timeouts are using the 2-arg convert rather than toMillis or toNanos so I've changed those too. The update webrev is here: > > http://cr.openjdk.java.net/~alanb/8222774/1/webrev/index.html The changes versus the previous version look OK to me. Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From claes.redestad at oracle.com Wed Apr 24 07:52:41 2019 From: claes.redestad at oracle.com (Claes Redestad) Date: Wed, 24 Apr 2019 09:52:41 +0200 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> <72cb59c0-e977-d297-589f-80f7d1a417e4@oracle.com> <1f20a70c-3d35-5005-12d4-d5a6853e4549@oracle.com> Message-ID: Hi Lance, On 2019-04-23 19:23, Lance Andersen wrote: > Hi Claes, > > The additional updates look fine. :-) thanks! I'm running some pre-integration tests and will wait a bit for any further feedback before push. /Claes From Alan.Bateman at oracle.com Wed Apr 24 13:15:40 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 24 Apr 2019 14:15:40 +0100 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: <1f20a70c-3d35-5005-12d4-d5a6853e4549@oracle.com> References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> <72cb59c0-e977-d297-589f-80f7d1a417e4@oracle.com> <1f20a70c-3d35-5005-12d4-d5a6853e4549@oracle.com> Message-ID: <2cfe48b3-1f19-7359-73fd-04bfde7867fc@oracle.com> On 23/04/2019 12:42, Claes Redestad wrote: > > Based on what it does, ExChannelCloser (which I restored from its pre- > existing state) should probably be named ExistingChannelCloser. > > I've collected the feedback from you, Christoph and Lance into this > new webrev: http://cr.openjdk.java.net/~redestad/8222532/open.01/ > > Incremental: http://cr.openjdk.java.net/~redestad/8222532/open.00_01/ I think this looks okay, at least to restore the previous behavior but I think we need to create a new bug to do a big cleanup to replace all of this (the previous existing commands with "ex-channel", "ex-path" are confusing so lots of renaming and cleanup needed I think). -Alan From claes.redestad at oracle.com Wed Apr 24 13:26:00 2019 From: claes.redestad at oracle.com (Claes Redestad) Date: Wed, 24 Apr 2019 15:26:00 +0200 Subject: RFR: 8222532: (zipfs) Performance regression when writing ZipFileSystem entries in parallel In-Reply-To: <2cfe48b3-1f19-7359-73fd-04bfde7867fc@oracle.com> References: <74d67729-fbea-2b5a-a840-d7abc2067682@oracle.com> <72cb59c0-e977-d297-589f-80f7d1a417e4@oracle.com> <1f20a70c-3d35-5005-12d4-d5a6853e4549@oracle.com> <2cfe48b3-1f19-7359-73fd-04bfde7867fc@oracle.com> Message-ID: On 2019-04-24 15:15, Alan Bateman wrote: > On 23/04/2019 12:42, Claes Redestad wrote: >> >> Based on what it does, ExChannelCloser (which I restored from its pre- >> existing state) should probably be named ExistingChannelCloser. >> >> I've collected the feedback from you, Christoph and Lance into this >> new webrev: http://cr.openjdk.java.net/~redestad/8222532/open.01/ >> >> Incremental: http://cr.openjdk.java.net/~redestad/8222532/open.00_01/ > I think this looks okay, at least to restore the previous behavior but I > think we need to create a new bug to do a big cleanup to replace all of > this (the previous existing commands with "ex-channel", "ex-path" are > confusing so lots of renaming and cleanup needed I think). Thanks Alan, I'll file an RFE to re-examine and clean things up. /Claes From brian.burkhalter at oracle.com Wed Apr 24 23:11:25 2019 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 24 Apr 2019 16:11:25 -0700 Subject: 8220793: (fs) No support for changing modification time of symlink In-Reply-To: References: <94054704-7CD6-4FE8-BE86-D17145C06E97@oracle.com> <6044b47b-74ff-1679-6b83-8cc1d71bcc39@oracle.com> Message-ID: > On Apr 22, 2019, at 11:19 AM, Brian Burkhalter wrote: > >> On Apr 22, 2019, at 12:36 AM, Alan Bateman > wrote: >> >> On 19/04/2019 01:24, Brian Burkhalter wrote: >>> For issue [1] please review the patch [2]. It implements setting the access and modification times of an unfollowed symbolic link on Linux and macOS. >>> >>> The patch uses the library function lutimes(3). The supported Unix platforms for lutimes() appear to be Linux and the BSDs, but neither Solaris (verified) nor AIX. Therefore the native function lutimes0() is a no-op for Unix platforms other than Linux and the BSDs. >>> >> I'm in two minds on this, mostly because it should be very rare to want to change the the time stamps of a symbolic link. >> >> In any case, can you double check setTimes - I assume it should not attempt to open the file when lutimes is available and the file is a sym link. > > It will only if just one of the times is being modified (lines 99-111). > > Also I neglected to put it in the retry section (lines 136-148) and that should likely be added. I added lutimes() to the retry section (lines 140-141) [1, 2]. Thanks, Brian [1] http://cr.openjdk.java.net/~bpb/8220793/webrev.00-01/ [2] http://cr.openjdk.java.net/~bpb/8220793/webrev.01/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adinn at redhat.com Thu Apr 25 16:34:31 2019 From: adinn at redhat.com (Andrew Dinn) Date: Thu, 25 Apr 2019 17:34:31 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> <9218d37f-9820-9216-b125-8f3bc2d19e3d@redhat.com> <453045f6-0259-e098-8211-0187428adb9f@oracle.com> Message-ID: <40cb2223-6d5a-4262-6293-2f187f578735@redhat.com> Also, here is a new webrev including the updated implementations for mappingAddress/Offset/Length as described below JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.02 regards, Andrew Dinn ----------- On 23/04/2019 17:15, Andrew Dinn wrote: > Hi Alan, > > Thanks for looking at this. > > On 22/04/2019 16:52, Alan Bateman wrote: >> The calculation to compute the page aligned address and then the >> distance from there to the end of the region is tricky to get right. I >> think you have it right, I'm just wondering if we could leverage the >> existing mappingOffset/mappingAddress methods so that we have only one >> place to audit. For example, suppose mappingOffset is changed to take an >> index. I think this would reduce the new force method down to: >> >> long offset = mappingOffset(index); >> force0(fd, mappingAddress(offset), offset + length); > > I like this idea but I'd like to make sure I am clear about exactly how > it ought to work. If you will indulge me I will work through this by > describing first the status quo (as I perceive it) and then what my > change attempts before trying to reconcile the two. > > Currently: > > There are 4 key locations of interest in the mapped byte range > > [... map_base ... address ... address+limit ... address+capacity ...] > | | > +----- page size -----+ > > where: > - addresses to the left are numerically (i.e. as longs) lower than > those to the right > - map_base is aligned to page size -- it is obtained by rounding down > address to a page boundary > - consequently (address - map_base) is less than page size > > n.b. the mmapped region may or may not commence at map_base but it will > always include map_base. Also, it will include bytes at all subsequent > addresses up to (address + capacity). Whether it includes or extends > beyond (address + capacity) is not really important. > > mappingOffset() computes (address - map_base): > > int ps = Bits.pageSize(); > long offset = address % ps; > return (offset >= 0) ? offset : (ps + offset); > > The jiggery-pokery with ? : works round the rather unhelpful behaviour > of mod wrt negative longs. > > I think it could, equivalently, compute it directly as: > > int ps = Bits.pageSize(); > long map_base = (address & ~(ps - 1)); > return address - map_base; > > mappingAddress(mappingOffset) converts a previously retrieved > mappingOffset back to the map base: > > return (address - mappingOffset) > == (address - (address - map_base)) > == map_base > > > mappingLength(mappingOffset) computes ((address + capacity) - map_base) > i.e. the byte count from start of page map_base to the last > (potentially) /writeable/ buffer byte. > > return capacity() + mappingOffset > == capacity + (address - map_base) > == (address + capacity) - map_base > > arguably this method could just compute ((address + length) - map_base) > i.e. the byte count from map_base to the last /written/ byte -- but that > makes no real difference when performing a file-based flush. > > The call to force0 is made as > > force0(fd, mappingAddress(offset), mappingLength(offset)) > > which is equivalent to > > force0(fd, map_base, capacity + (address - map_base)) > > > My updated code: > > When looking at an indexed location there are now 7 key locations of > interest in the mapped byte range > > [... m_base ... a ... i_base ... a+i ... a+i+ln ... a+lim, a+cap ...] > | | | | > +- page size -+ +- page size -+ > > > where: > - for brevity, map_base is abbreviated to m_base, address to a, > index_base to i_base, index to i, length to ln, limit to lim > and capacity to cap > - index is identifies the start of the region to be flushed > - length is the length of the region to be flushed > - (address + index) is the address of the byte at offset index > - index_base is the largest page aligned address below (address + idx) > i.e. it is obtained by rounding down the first byte of the flush > region to a page boundary > > index_base is computed as follows > > int ps = Bits.pageSize(); > long index_base = ((address + index) & ~(ps - 1)); > > My code is calling > > force0(fd, index_base, length + ((address + index) - i_base)) > > i.e. it the flushed range starts at the nearest page boundary below or > equal to (address + index) and includes the original length bytes plus > the intervening bytes between that page boundary and address. > > When this is considered by analogy with the previous call then three > things stand out: > > 1) in place of map_base for the first argument we have index_base > 2) in place of the extra included intervening bytes between map_base > and address we have the extra included intervening bytes between > index_base and address + index > 3) in place of capacity (default) demarcated bytes to be flushed we > have length user demarcated bytes to be flushed > > So, to retain those parallels detailed in 1, 2 and 3 we need > mappingOffset and mappingAddress to be overloaded and the originals > rerouted as follows > > mappingOffset(index) > { > int ps = Bits.pageSize(); > long index_address = address + index; > long index_base = (index_address & ~(ps - 1)); > return index_address - index_base; > } > > mappingOffset() => mappingOffset(0) > > mappingAddress(mappingOffset, index) > { > long index_address = address + index; > return index_address - mappingOffset > == (index_address - (index_address - index_base)) > == index_base > } > > mappingAddress(mappingOffset) => mappingAddress(mappingOffset, 0) > > mappingLength(mappingOffset, length) > { > return length + mappingOffset > } > > mappingLength(mappingOffset) => > mappingLength(mappingOffset, (long)capacity()) > > Does that look correct? If so I will prepare a new webrev accordingly. > >> I don't expect any issues on Windows but I will test it to make sure. > Thank you. That would be very helpful. > > regards, > > > Andrew Dinn > ----------- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in England and Wales under Company Registration No. 03798903 > Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander > From christoph.langer at sap.com Fri Apr 26 12:35:51 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 26 Apr 2019 12:35:51 +0000 Subject: RFR (S): 8223015: Cleanups for zipfs tests Message-ID: Hi, please help reviewing these cleanups to the zipfs tests. In detail: - Fix warnings in test/jdk/jdk/nio/zipfs/Demo.java (not a real testcase but Demo coding) - change some calls of Runtime.version().major() to Runtime.version().feature() (the former is deprecated) - change occurences of new Integer() to Integer.valueOf() - change occurences of class.newInstance() to class.getDeclaredConstructor().newInstance() - remove unused variables - move test/jdk/jdk/nio/zipfs/jarfs/JFSTester.java to test/jdk/jdk/nio/zipfs/JFSTester.java (not using the jarfs subfolder). There is no need for this specific subfolder, given that MultiReleaseJarTest.java neither isn't in the jarfs subfolder Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8223015.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8223015 I verified that the tests still succeed with the change. Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From claes.redestad at oracle.com Fri Apr 26 12:44:24 2019 From: claes.redestad at oracle.com (Claes Redestad) Date: Fri, 26 Apr 2019 14:44:24 +0200 Subject: RFR (S): 8223015: Cleanups for zipfs tests In-Reply-To: References: Message-ID: Looks good to me. I took some extra care to check that use of new Integer in MultiReleaseJarTest wasn't intentional but that doesn't seem to be the case here. Thanks! /Claes On 2019-04-26 14:35, Langer, Christoph wrote: > Hi, > > please help reviewing these cleanups to the zipfs tests. In detail: > > - Fix warnings in test/jdk/jdk/nio/zipfs/Demo.java (not a real testcase > but Demo coding) > - change some calls of Runtime.version().major() to > Runtime.version().feature() (the former is deprecated) > - change occurences of new Integer() to Integer.valueOf() > - change occurences of class.newInstance() to > class.getDeclaredConstructor().newInstance() > - remove unused variables > - move test/jdk/jdk/nio/zipfs/jarfs/JFSTester.java to > test/jdk/jdk/nio/zipfs/JFSTester.java (not using the jarfs subfolder). > There is no need for this specific subfolder, given that > MultiReleaseJarTest.java neither isn't in the jarfs subfolder > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8223015.0/ > > Bug: https://bugs.openjdk.java.net/browse/JDK-8223015 > > I verified that the tests still succeed with the change. > > Thanks > > Christoph > From christoph.langer at sap.com Fri Apr 26 12:46:33 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 26 Apr 2019 12:46:33 +0000 Subject: RFR (S): 8223015: Cleanups for zipfs tests In-Reply-To: References: Message-ID: Thank you, Claes. > -----Original Message----- > From: Claes Redestad > Sent: Freitag, 26. April 2019 14:44 > To: Langer, Christoph ; core-libs- > dev at openjdk.java.net; nio-dev at openjdk.java.net > Subject: Re: RFR (S): 8223015: Cleanups for zipfs tests > > Looks good to me. > > I took some extra care to check that use of new Integer in > MultiReleaseJarTest wasn't intentional but that doesn't seem to be the > case here. > > Thanks! > > /Claes > > On 2019-04-26 14:35, Langer, Christoph wrote: > > Hi, > > > > please help reviewing these cleanups to the zipfs tests. In detail: > > > > - Fix warnings in test/jdk/jdk/nio/zipfs/Demo.java (not a real testcase > > but Demo coding) > > - change some calls of Runtime.version().major() to > > Runtime.version().feature() (the former is deprecated) > > - change occurences of new Integer() to Integer.valueOf() > > - change occurences of class.newInstance() to > > class.getDeclaredConstructor().newInstance() > > - remove unused variables > > - move test/jdk/jdk/nio/zipfs/jarfs/JFSTester.java to > > test/jdk/jdk/nio/zipfs/JFSTester.java (not using the jarfs subfolder). > > There is no need for this specific subfolder, given that > > MultiReleaseJarTest.java neither isn't in the jarfs subfolder > > > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8223015.0/ > > > > Bug: https://bugs.openjdk.java.net/browse/JDK-8223015 > > > > I verified that the tests still succeed with the change. > > > > Thanks > > > > Christoph > > From Alan.Bateman at oracle.com Fri Apr 26 14:46:01 2019 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Fri, 26 Apr 2019 15:46:01 +0100 Subject: RFR : 8221696: MappedByteBuffer.force method to specify range In-Reply-To: <40cb2223-6d5a-4262-6293-2f187f578735@redhat.com> References: <2c9099f0-d95d-425a-c033-b22ad5c25231@redhat.com> <4ad97801-218f-2fa5-e8b5-3f14347c4320@oracle.com> <9218d37f-9820-9216-b125-8f3bc2d19e3d@redhat.com> <453045f6-0259-e098-8211-0187428adb9f@oracle.com> <40cb2223-6d5a-4262-6293-2f187f578735@redhat.com> Message-ID: On 25/04/2019 17:34, Andrew Dinn wrote: > Also, here is a new webrev including the updated implementations for > mappingAddress/Offset/Length as described below > > JIRA: https://bugs.openjdk.java.net/browse/JDK-8221696 > webrev: http://cr.openjdk.java.net/~adinn/8221696/webrev.02 > This looks right and I agree with your notes on negative addresses. As its tricky to get right then it would be good to get another set of eyes on this. A minor nit is that we don't usually use underscores in variable names here, it would be baseAddress rather than base_address, etc. -Alan From christoph.langer at sap.com Fri Apr 26 15:37:25 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 26 Apr 2019 15:37:25 +0000 Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 In-Reply-To: References: Message-ID: Hi, ping ?? I?ve rebased this change after the latest zipfs updates. I also made some further modifications. Apart from minor things, I modified the constructors of the ZipFileSystem.Entry class and I also made some post JDK-8222440 cleanup. Please find the new webrev here: http://cr.openjdk.java.net/~clanger/webrevs/8222276.1/ The jtreg tests for zipfs are all passing. Thanks Christoph From: Langer, Christoph Sent: Donnerstag, 11. April 2019 16:06 To: core-libs-dev ; nio-dev ; Alan Bateman ; Lance Andersen Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 Hi, working on JDK-8213031 to add posix file permission support for the zipfs, I thought it makes sense to factor out some of the changes into a separate change. Those changes reorganize and clean up some parts of the code which I think makes sense in any case. Please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8222276 Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8222276.0/ In detail: The main change is to the hierarchy of inner classes ZipFileSystem.IndexNode and ZipFileSystem.Entry (the latter extends the former). Both classes are static classes currently. It makes sense, though, to have those associated with the ZipFileSystem they belong to. To do that, I need to add a static superclass named ZFSNode which only holds a node name and its hashcode. This class needs to exist because of the lookup implementation to find specific nodes of tze ZipFileSystem in its node map. Then the classes IndexNode extends ZFSNode and Entry extends IndexNode can be made non-static. Further changes are: Improve error handling for ZipFileAttributeView::get methods improve handling for zip64 attribute/version minor clenaups such as adding @Override annotations, whitespace fixes, private method visibility etc. Furthermore, there is some suspicious coding in JarFileSystem:: createVersionedLinks where a temporary key node is inserted into the alias map. However, this is a singleton instance which always gets new values. I?ll open a separate bug for that and I?ll try to create a test case which demonstrates an issue with the current code. I ran the zipfs jtreg tests successfully and will put it into our internal system as well as run it through jdk-submit. Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance.andersen at oracle.com Fri Apr 26 16:25:37 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 26 Apr 2019 12:25:37 -0400 Subject: RFR (S): 8223015: Cleanups for zipfs tests In-Reply-To: References: Message-ID: <5B6F2470-2E7C-4E82-B90B-7D5A4A80F809@oracle.com> Hi Christoph, Thank you for doing this. Overall the changes look good. A few minor thoughts ZIpFSTester.java - 537: Given you removed the method variable, I would tweak the printf string to make it a bit clearer that what the output its. JFSTester.java and MultiReleaseJarTest.java - I had the opposite feeling that having a jarfs subfolder adds clarity and moving MulltiReleaseJarTest.java might be the better way to go. It helps separate specific tests which are using the JarFileSystem from the ZipFileSystem especially as we add more, keeping things cleaner. Best Lance > On Apr 26, 2019, at 8:35 AM, Langer, Christoph wrote: > > Hi, > > please help reviewing these cleanups to the zipfs tests. In detail: > > - Fix warnings in test/jdk/jdk/nio/zipfs/Demo.java (not a real testcase but Demo coding) > - change some calls of Runtime.version().major() to Runtime.version().feature() (the former is deprecated) > - change occurences of new Integer() to Integer.valueOf() > - change occurences of class.newInstance() to class.getDeclaredConstructor().newInstance() > - remove unused variables > - move test/jdk/jdk/nio/zipfs/jarfs/JFSTester.java to test/jdk/jdk/nio/zipfs/JFSTester.java (not using the jarfs subfolder). There is no need for this specific subfolder, given that MultiReleaseJarTest.java neither isn't in the jarfs subfolder > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8223015.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8223015 > > I verified that the tests still succeed with the change. > > Thanks > Christoph > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From lance.andersen at oracle.com Fri Apr 26 16:44:36 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 26 Apr 2019 12:44:36 -0400 Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 In-Reply-To: References: Message-ID: On my todo list, will get to it either later today or over the weekend. Thank you for the reminder :-) > On Apr 26, 2019, at 11:37 AM, Langer, Christoph wrote: > > Hi, > > ping ?? > > I?ve rebased this change after the latest zipfs updates. I also made some further modifications. Apart from minor things, I modified the constructors of the ZipFileSystem.Entry class and I also made some post JDK-8222440 cleanup. > > Please find the new webrev here: http://cr.openjdk.java.net/~clanger/webrevs/8222276.1/ > > The jtreg tests for zipfs are all passing. > > Thanks > Christoph > > From: Langer, Christoph > Sent: Donnerstag, 11. April 2019 16:06 > To: core-libs-dev ; nio-dev ; Alan Bateman ; Lance Andersen > Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 > > Hi, > > working on JDK-8213031 to add posix file permission support for the zipfs, I thought it makes sense to factor out some of the changes into a separate change. Those changes reorganize and clean up some parts of the code which I think makes sense in any case. > > Please review: > Bug: https://bugs.openjdk.java.net/browse/JDK-8222276 > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8222276.0/ > > In detail: > The main change is to the hierarchy of inner classes ZipFileSystem.IndexNode and ZipFileSystem.Entry (the latter extends the former). Both classes are static classes currently. It makes sense, though, to have those associated with the ZipFileSystem they belong to. > To do that, I need to add a static superclass named ZFSNode which only holds a node name and its hashcode. This class needs to exist because of the lookup implementation to find specific nodes of tze ZipFileSystem in its node map. Then the classes IndexNode extends ZFSNode and Entry extends IndexNode can be made non-static. > > Further changes are: > Improve error handling for ZipFileAttributeView::get methods > improve handling for zip64 attribute/version > minor clenaups such as adding @Override annotations, whitespace fixes, private method visibility etc. > > Furthermore, there is some suspicious coding in JarFileSystem:: createVersionedLinks where a temporary key node is inserted into the alias map. However, this is a singleton instance which always gets new values. I?ll open a separate bug for that and I?ll try to create a test case which demonstrates an issue with the current code. > > I ran the zipfs jtreg tests successfully and will put it into our internal system as well as run it through jdk-submit. > > Thanks > Christoph Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From christoph.langer at sap.com Fri Apr 26 16:50:38 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 26 Apr 2019 16:50:38 +0000 Subject: RFR (S): 8223015: Cleanups for zipfs tests In-Reply-To: <5B6F2470-2E7C-4E82-B90B-7D5A4A80F809@oracle.com> References: <5B6F2470-2E7C-4E82-B90B-7D5A4A80F809@oracle.com> Message-ID: Hi Lance, thanks for reviewing. As for ZipFSTester: the printf is anyway commented out, probably only used for debugging. Removing the variable was just because the IDE said it's unused. I'd rather leave it as is. As for the jarfs tests: OK, it probably makes sense to separate these out into a subfolder. So I'll move MultiReleaseJarTest to the subfolder instead of moving JFSTester. OK for you if I push this without another webrev next week? Best regards Christoph From: Lance Andersen Sent: Freitag, 26. April 2019 18:26 To: Langer, Christoph Cc: core-libs-dev at openjdk.java.net; nio-dev at openjdk.java.net Subject: Re: RFR (S): 8223015: Cleanups for zipfs tests Hi Christoph, Thank you for doing this. Overall the changes look good. A few minor thoughts ZIpFSTester.java - 537: Given you removed the method variable, I would tweak the printf string to make it a bit clearer that what the output its. JFSTester.java and MultiReleaseJarTest.java - I had the opposite feeling that having a jarfs subfolder adds clarity and moving MulltiReleaseJarTest.java might be the better way to go. It helps separate specific tests which are using the JarFileSystem from the ZipFileSystem especially as we add more, keeping things cleaner. Best Lance On Apr 26, 2019, at 8:35 AM, Langer, Christoph > wrote: Hi, please help reviewing these cleanups to the zipfs tests. In detail: - Fix warnings in test/jdk/jdk/nio/zipfs/Demo.java (not a real testcase but Demo coding) - change some calls of Runtime.version().major() to Runtime.version().feature() (the former is deprecated) - change occurences of new Integer() to Integer.valueOf() - change occurences of class.newInstance() to class.getDeclaredConstructor().newInstance() - remove unused variables - move test/jdk/jdk/nio/zipfs/jarfs/JFSTester.java to test/jdk/jdk/nio/zipfs/JFSTester.java (not using the jarfs subfolder). There is no need for this specific subfolder, given that MultiReleaseJarTest.java neither isn't in the jarfs subfolder Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8223015.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8223015 I verified that the tests still succeed with the change. Thanks Christoph [cid:image001.gif at 01D4FC60.EDECDC10] Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 658 bytes Desc: image001.gif URL: From christoph.langer at sap.com Fri Apr 26 16:51:16 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 26 Apr 2019 16:51:16 +0000 Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 In-Reply-To: References: Message-ID: Great, thank you. This one is a bit larger and will probably need a few iterations? From: Lance Andersen Sent: Freitag, 26. April 2019 18:45 To: Langer, Christoph Cc: core-libs-dev ; nio-dev ; Alan Bateman Subject: Re: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 On my todo list, will get to it either later today or over the weekend. Thank you for the reminder :-) On Apr 26, 2019, at 11:37 AM, Langer, Christoph > wrote: Hi, ping ?? I?ve rebased this change after the latest zipfs updates. I also made some further modifications. Apart from minor things, I modified the constructors of the ZipFileSystem.Entry class and I also made some post JDK-8222440 cleanup. Please find the new webrev here: http://cr.openjdk.java.net/~clanger/webrevs/8222276.1/ The jtreg tests for zipfs are all passing. Thanks Christoph From: Langer, Christoph Sent: Donnerstag, 11. April 2019 16:06 To: core-libs-dev >; nio-dev >; Alan Bateman >; Lance Andersen > Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 Hi, working on JDK-8213031 to add posix file permission support for the zipfs, I thought it makes sense to factor out some of the changes into a separate change. Those changes reorganize and clean up some parts of the code which I think makes sense in any case. Please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8222276 Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8222276.0/ In detail: The main change is to the hierarchy of inner classes ZipFileSystem.IndexNode and ZipFileSystem.Entry (the latter extends the former). Both classes are static classes currently. It makes sense, though, to have those associated with the ZipFileSystem they belong to. To do that, I need to add a static superclass named ZFSNode which only holds a node name and its hashcode. This class needs to exist because of the lookup implementation to find specific nodes of tze ZipFileSystem in its node map. Then the classes IndexNode extends ZFSNode and Entry extends IndexNode can be made non-static. Further changes are: Improve error handling for ZipFileAttributeView::get methods improve handling for zip64 attribute/version minor clenaups such as adding @Override annotations, whitespace fixes, private method visibility etc. Furthermore, there is some suspicious coding in JarFileSystem:: createVersionedLinks where a temporary key node is inserted into the alias map. However, this is a singleton instance which always gets new values. I?ll open a separate bug for that and I?ll try to create a test case which demonstrates an issue with the current code. I ran the zipfs jtreg tests successfully and will put it into our internal system as well as run it through jdk-submit. Thanks Christoph [cid:image001.gif at 01D4FC61.06443E70] Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 658 bytes Desc: image001.gif URL: From lance.andersen at oracle.com Fri Apr 26 16:54:17 2019 From: lance.andersen at oracle.com (Lance Andersen) Date: Fri, 26 Apr 2019 12:54:17 -0400 Subject: RFR (S): 8223015: Cleanups for zipfs tests In-Reply-To: References: <5B6F2470-2E7C-4E82-B90B-7D5A4A80F809@oracle.com> Message-ID: Hi Christoph, > On Apr 26, 2019, at 12:50 PM, Langer, Christoph wrote: > > Hi Lance, > > thanks for reviewing. :-) > > As for ZipFSTester: the printf is anyway commented out, probably only used for debugging. Removing the variable was just because the IDE said it?s unused. I?d rather leave it as is. Well, i would tweak it or remove it. > > As for the jarfs tests: OK, it probably makes sense to separate these out into a subfolder. So I?ll move MultiReleaseJarTest to the subfolder instead of moving JFSTester. sounds like a plan > > OK for you if I push this without another webrev next week? Yes that is fine :-) Have a good weekend! Best Lance > > Best regards > Christoph > > From: Lance Andersen > > Sent: Freitag, 26. April 2019 18:26 > To: Langer, Christoph > > Cc: core-libs-dev at openjdk.java.net ; nio-dev at openjdk.java.net > Subject: Re: RFR (S): 8223015: Cleanups for zipfs tests > > Hi Christoph, > > Thank you for doing this. Overall the changes look good. > > A few minor thoughts > > ZIpFSTester.java - 537: Given you removed the method variable, I would tweak the printf string to make it a bit clearer that what the output its. > > JFSTester.java and MultiReleaseJarTest.java - I had the opposite feeling that having a jarfs subfolder adds clarity and moving MulltiReleaseJarTest.java might be the better way to go. It helps separate specific tests which are using the JarFileSystem from the ZipFileSystem especially as we add more, keeping things cleaner. > > Best > Lance > > > On Apr 26, 2019, at 8:35 AM, Langer, Christoph > wrote: > > Hi, > > please help reviewing these cleanups to the zipfs tests. In detail: > > - Fix warnings in test/jdk/jdk/nio/zipfs/Demo.java (not a real testcase but Demo coding) > - change some calls of Runtime.version().major() to Runtime.version().feature() (the former is deprecated) > - change occurences of new Integer() to Integer.valueOf() > - change occurences of class.newInstance() to class.getDeclaredConstructor().newInstance() > - remove unused variables > - move test/jdk/jdk/nio/zipfs/jarfs/JFSTester.java to test/jdk/jdk/nio/zipfs/JFSTester.java (not using the jarfs subfolder). There is no need for this specific subfolder, given that MultiReleaseJarTest.java neither isn't in the jarfs subfolder > > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8223015.0/ > Bug: https://bugs.openjdk.java.net/browse/JDK-8223015 > > I verified that the tests still succeed with the change. > > Thanks > Christoph > > > > > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 > Oracle Java Engineering > 1 Network Drive > Burlington, MA 01803 > Lance.Andersen at oracle.com > > > > Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: oracle_sig_logo.gif Type: image/gif Size: 658 bytes Desc: not available URL: From christoph.langer at sap.com Fri Apr 26 17:04:16 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Fri, 26 Apr 2019 17:04:16 +0000 Subject: RFR (S): 8223015: Cleanups for zipfs tests In-Reply-To: References: <5B6F2470-2E7C-4E82-B90B-7D5A4A80F809@oracle.com> Message-ID: Hi Lance, ok, then I?ll remove the commented printf in ZipFSTester altogether. In case anybody debugs this again, he or she will probably add their own printfs anyway. Have a nice weekend, too ?? /Christoph From: Lance Andersen Sent: Freitag, 26. April 2019 18:54 To: Langer, Christoph Cc: core-libs-dev at openjdk.java.net; nio-dev at openjdk.java.net Subject: Re: RFR (S): 8223015: Cleanups for zipfs tests Hi Christoph, On Apr 26, 2019, at 12:50 PM, Langer, Christoph > wrote: Hi Lance, thanks for reviewing. :-) As for ZipFSTester: the printf is anyway commented out, probably only used for debugging. Removing the variable was just because the IDE said it?s unused. I?d rather leave it as is. Well, i would tweak it or remove it. As for the jarfs tests: OK, it probably makes sense to separate these out into a subfolder. So I?ll move MultiReleaseJarTest to the subfolder instead of moving JFSTester. sounds like a plan OK for you if I push this without another webrev next week? Yes that is fine :-) Have a good weekend! Best Lance Best regards Christoph From: Lance Andersen > Sent: Freitag, 26. April 2019 18:26 To: Langer, Christoph > Cc: core-libs-dev at openjdk.java.net; nio-dev at openjdk.java.net Subject: Re: RFR (S): 8223015: Cleanups for zipfs tests Hi Christoph, Thank you for doing this. Overall the changes look good. A few minor thoughts ZIpFSTester.java - 537: Given you removed the method variable, I would tweak the printf string to make it a bit clearer that what the output its. JFSTester.java and MultiReleaseJarTest.java - I had the opposite feeling that having a jarfs subfolder adds clarity and moving MulltiReleaseJarTest.java might be the better way to go. It helps separate specific tests which are using the JarFileSystem from the ZipFileSystem especially as we add more, keeping things cleaner. Best Lance On Apr 26, 2019, at 8:35 AM, Langer, Christoph > wrote: Hi, please help reviewing these cleanups to the zipfs tests. In detail: - Fix warnings in test/jdk/jdk/nio/zipfs/Demo.java (not a real testcase but Demo coding) - change some calls of Runtime.version().major() to Runtime.version().feature() (the former is deprecated) - change occurences of new Integer() to Integer.valueOf() - change occurences of class.newInstance() to class.getDeclaredConstructor().newInstance() - remove unused variables - move test/jdk/jdk/nio/zipfs/jarfs/JFSTester.java to test/jdk/jdk/nio/zipfs/JFSTester.java (not using the jarfs subfolder). There is no need for this specific subfolder, given that MultiReleaseJarTest.java neither isn't in the jarfs subfolder Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8223015.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8223015 I verified that the tests still succeed with the change. Thanks Christoph Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com [cid:image001.gif at 01D4FC62.D708F4F0] Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen at oracle.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 658 bytes Desc: image001.gif URL: From christoph.langer at sap.com Tue Apr 30 09:54:01 2019 From: christoph.langer at sap.com (Langer, Christoph) Date: Tue, 30 Apr 2019 09:54:01 +0000 Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 In-Reply-To: References: Message-ID: Hi, it turned out that there was a regression in my last version. I needed to fix some initializations in the constructor of Entry. I also added some other fixes found by the IDE?s code analysis to ZipFileSystem.java. By accident, I updated the last webrev in-place. So, please find my most current version here: http://cr.openjdk.java.net/~clanger/webrevs/8222276.1/ Now it passes all testing. Thank you Christoph From: Langer, Christoph Sent: Freitag, 26. April 2019 17:37 To: core-libs-dev ; nio-dev ; Alan Bateman ; Lance Andersen Subject: RE: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 Hi, ping ?? I?ve rebased this change after the latest zipfs updates. I also made some further modifications. Apart from minor things, I modified the constructors of the ZipFileSystem.Entry class and I also made some post JDK-8222440 cleanup. Please find the new webrev here: http://cr.openjdk.java.net/~clanger/webrevs/8222276.1/ The jtreg tests for zipfs are all passing. Thanks Christoph From: Langer, Christoph Sent: Donnerstag, 11. April 2019 16:06 To: core-libs-dev >; nio-dev >; Alan Bateman >; Lance Andersen > Subject: RFR: 8222276: (zipfs) Refactoring and cleanups to prepare for JDK-8213031 Hi, working on JDK-8213031 to add posix file permission support for the zipfs, I thought it makes sense to factor out some of the changes into a separate change. Those changes reorganize and clean up some parts of the code which I think makes sense in any case. Please review: Bug: https://bugs.openjdk.java.net/browse/JDK-8222276 Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8222276.0/ In detail: The main change is to the hierarchy of inner classes ZipFileSystem.IndexNode and ZipFileSystem.Entry (the latter extends the former). Both classes are static classes currently. It makes sense, though, to have those associated with the ZipFileSystem they belong to. To do that, I need to add a static superclass named ZFSNode which only holds a node name and its hashcode. This class needs to exist because of the lookup implementation to find specific nodes of tze ZipFileSystem in its node map. Then the classes IndexNode extends ZFSNode and Entry extends IndexNode can be made non-static. Further changes are: Improve error handling for ZipFileAttributeView::get methods improve handling for zip64 attribute/version minor clenaups such as adding @Override annotations, whitespace fixes, private method visibility etc. Furthermore, there is some suspicious coding in JarFileSystem:: createVersionedLinks where a temporary key node is inserted into the alias map. However, this is a singleton instance which always gets new values. I?ll open a separate bug for that and I?ll try to create a test case which demonstrates an issue with the current code. I ran the zipfs jtreg tests successfully and will put it into our internal system as well as run it through jdk-submit. Thanks Christoph -------------- next part -------------- An HTML attachment was scrubbed... URL: