From vyom.tewari at oracle.com Fri Dec 1 07:13:23 2017 From: vyom.tewari at oracle.com (vyom tewari) Date: Fri, 1 Dec 2017 12:43:23 +0530 Subject: RFR:8190843 can not set/get extendedOptions to ServerSocket In-Reply-To: <88BCFD69-DD84-4DC7-8D60-5859FF0D80FB@oracle.com> References: <246540ed-298b-81d7-1ea5-3b41b2436b62@oracle.com> <031a8609-3476-1159-f537-1cf974fe5729@oracle.com> <4f5def6b-5809-b11f-a93a-d934a4bfd31d@oracle.com> <88BCFD69-DD84-4DC7-8D60-5859FF0D80FB@oracle.com> Message-ID: <6b499005-f093-8751-ba38-5c16a4490e08@oracle.com> Hi Chris, Thanks for review, while my testing i discovered issue in the way we handle extended? socket options and standard socket options. I fixed it and updated one test as well. I removed one redundant "if check" which i think not required. JPRT is clean with the changed code. Please find the latest webrev(http://cr.openjdk.java.net/~vtewari/8190843/webrev0.2/index.html). Thanks, Vyom On Thursday 23 November 2017 06:04 PM, Chris Hegarty wrote: >> On 23 Nov 2017, at 09:20, vyom tewari wrote: >> >> Hi Chris, >> >> Thanks for the review, please find the latest webrev(http://cr.openjdk.java.net/~vtewari/8190843/webrev0.1/index.html) where i modified the code to take care of SO_FLOW for Solaris. > I think this is fine Vyom, thanks. > > -Chris. > From david.lloyd at redhat.com Fri Dec 1 18:12:28 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Fri, 1 Dec 2017 12:12:28 -0600 Subject: Parsing too strict in java.net.URI? In-Reply-To: <3d37643e-2ce6-3d30-5ec8-1b80be540343@oracle.com> References: <93c61de4-6843-a66b-eef8-456aa4e1238b@redhat.com> <3d37643e-2ce6-3d30-5ec8-1b80be540343@oracle.com> Message-ID: On Mon, Nov 14, 2016 at 12:16 PM, Chris Hegarty wrote: > David, > > On 14/11/16 16:47, David M. Lloyd wrote: >> >> The following statement: >> >> URI uri = URI.create("local:"); >> >> fails with an exception like this: [...] >> However AFAICT scheme-only URIs are, while not strictly allowed by RFC >> 2396 [1], in common usage (for example, "about:" in web browsers). > > This seems to be allowable in the "more recent" RFC [2], that obsoletes > 2396, which of course java.net.URI does not, yet, support. There was an > effort to support 3986 a number of years ago, but it was not successful. > The desire to support, keep up with standards, etc, has caused us to > give consideration to supporting 3986, but as of yet nothing concrete. Have there been any recent developments, updates, or insights on this issue? I wonder if a gradual change to RFC 3986 support might be more appropriate: for example, start with *just* accepting empty SSP. If the compatibility requirements of java.net.URI are too stringent to allow for any change, then surely all that can be done is to deprecate & replace it, which would need no further delay that I can see. -- - DML From Roger.Riggs at oracle.com Sat Dec 2 02:39:23 2017 From: Roger.Riggs at oracle.com (Roger Riggs) Date: Fri, 1 Dec 2017 21:39:23 -0500 Subject: RFR:8190843 can not set/get extendedOptions to ServerSocket In-Reply-To: <246540ed-298b-81d7-1ea5-3b41b2436b62@oracle.com> References: <246540ed-298b-81d7-1ea5-3b41b2436b62@oracle.com> Message-ID: Hi Vyom, Looks fine Roger On 11/16/2017 4:03 AM, vyom tewari wrote: > > Hi All, > > Please review the small code change for the below issue. > > Webrev???? : > http://cr.openjdk.java.net/~vtewari/8190843/webrev0.0/index.html > > BugId??????? : https://bugs.openjdk.java.net/browse/JDK-8190843 > > In this code change, i removed the check(getSocket() == null) which > will always be true for ServerSocket as in case of server socket we > set "serverSocket" not "socket". > > Thanks, > > Vyom > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Sat Dec 2 09:17:26 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Sat, 2 Dec 2017 09:17:26 +0000 Subject: RFR:8190843 can not set/get extendedOptions to ServerSocket In-Reply-To: <6b499005-f093-8751-ba38-5c16a4490e08@oracle.com> References: <246540ed-298b-81d7-1ea5-3b41b2436b62@oracle.com> <031a8609-3476-1159-f537-1cf974fe5729@oracle.com> <4f5def6b-5809-b11f-a93a-d934a4bfd31d@oracle.com> <88BCFD69-DD84-4DC7-8D60-5859FF0D80FB@oracle.com> <6b499005-f093-8751-ba38-5c16a4490e08@oracle.com> Message-ID: <80123609-2B8E-41CE-BCCE-A1633AE749D9@oracle.com> > On 1 Dec 2017, at 07:13, vyom tewari wrote: > > Hi Chris, > > Thanks for review, while my testing i discovered issue in the way we handle extended socket options and standard socket options. I fixed it and updated one test as well. > > I removed one redundant "if check" which i think not required. JPRT is clean with the changed code. > > Please find the latest webrev(http://cr.openjdk.java.net/~vtewari/8190843/webrev0.2/index.html). Thanks Vyom, looks good. -Chris. From Roger.Riggs at Oracle.com Sat Dec 2 16:22:43 2017 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Sat, 2 Dec 2017 11:22:43 -0500 Subject: RFR:8190843 can not set/get extendedOptions to ServerSocket In-Reply-To: <80123609-2B8E-41CE-BCCE-A1633AE749D9@oracle.com> References: <246540ed-298b-81d7-1ea5-3b41b2436b62@oracle.com> <031a8609-3476-1159-f537-1cf974fe5729@oracle.com> <4f5def6b-5809-b11f-a93a-d934a4bfd31d@oracle.com> <88BCFD69-DD84-4DC7-8D60-5859FF0D80FB@oracle.com> <6b499005-f093-8751-ba38-5c16a4490e08@oracle.com> <80123609-2B8E-41CE-BCCE-A1633AE749D9@oracle.com> Message-ID: <8f2c16c5-2720-6c01-553c-bf35474cc992@Oracle.com> +1 On 12/2/2017 4:17 AM, Chris Hegarty wrote: >> On 1 Dec 2017, at 07:13, vyom tewari wrote: >> >> Hi Chris, >> >> Thanks for review, while my testing i discovered issue in the way we handle extended socket options and standard socket options. I fixed it and updated one test as well. >> >> I removed one redundant "if check" which i think not required. JPRT is clean with the changed code. >> >> Please find the latest webrev(http://cr.openjdk.java.net/~vtewari/8190843/webrev0.2/index.html). > Thanks Vyom, looks good. > > -Chris. From peter.firmstone at zeus.net.au Sat Dec 2 22:44:13 2017 From: peter.firmstone at zeus.net.au (Peter Firmstone) Date: Sun, 03 Dec 2017 08:44:13 +1000 Subject: Parsing too strict in java.net.URI? (David Lloyd) Message-ID: <5A232CBD.2010508@zeus.net.au> > Have there been any recent developments, updates, or insights on this > issue? I wonder if a gradual change to RFC 3986 support might be more > appropriate: for example, start with*just* accepting empty SSP. > > If the compatibility requirements of java.net.URI are too stringent to > allow for any change, then surely all that can be done is to deprecate > & replace it, which would need no further delay that I can see. A strict RFC 3896 implementation, modified from Apache Harmony's java.net.URI to support RFC3986. This is also in Apache River, the code has provenance, if the other authors are still contactable, you might get it donated as GPL. It doesn't implement Serializable, it does have method signatures identical to java.net.URI, so it's a drop in replacement in most cases. It has some additional methods, to fix illegal but commonly used URI forms. You could use it to test the impact of migrating to RFC3986, without having to implement it yourselves. https://github.com/pfirmstone/JGDMS/tree/trunk/JGDMS/jgdms-platform/src/main/java/org/apache/river/api/net Regards, Peter. From mark.reinhold at oracle.com Mon Dec 4 16:17:09 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 4 Dec 2017 08:17:09 -0800 (PST) Subject: JEP 321: HTTP Client (Standard) Message-ID: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> New JEP Candidate: http://openjdk.java.net/jeps/321 - Mark From david.lloyd at redhat.com Mon Dec 4 18:41:07 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 4 Dec 2017 12:41:07 -0600 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: On Mon, Dec 4, 2017 at 10:17 AM, wrote: > New JEP Candidate: http://openjdk.java.net/jeps/321 I have concerns. This will be the first public, NIO-based, asynchronous/non-blocking network protocol API introduced into the JDK proper, _ever_. First, I want to note that the API seems to bear no resemblance whatsoever with the asynchronous NIO.2 API. Now I'm no fan of that API as it stands for a couple of reasons, but nevertheless it calls into question either the validity of the HTTP API as it stands, or the scope of reusability of the NIO.2 API. With that items put aside: there are a wide variety of mature, non-blocking network protocol implementations out there, with literally thousands of years of experience distributed amongst their authors, maintainers, supporters, and communities, none of which were used as a model. There was, as far as I can see, no kind of study of existing non-blocking approaches in Java, and their strengths and weaknesses; there was no round table of engineers with experience in this field, talking about what works and what doesn't work. Some new, seemingly general-purpose, concepts are introduced by the code base, for example: ByteBufferReference, and ByteBufferPool. Are these strategies that should be promoted to NIO proper? If not, then are they _really_ right for this particular use case, particularly if, for some reason, a _second_ non-blocking network protocol API might be introduced some day, probably duplicating these concepts? Making this thing be the first real platform NIO-based asynchronous network protocol API, yet being completely removed from the previous NIO.2 asynchronous APIs and any other existing stable, mature API, should be done very carefully and deliberately, and perhaps most importantly, incrementally: first, establish a non-blocking byte stream API that makes sense generally, and bring that into NIO (NIO.3?); then, perhaps, enhancements to byte buffers to better support efficient pooling. By the time that point is reached, it is hopefully rapidly becoming obvious that this is not something that should be taken lightly. I believe that most third-party implementations are taken less lightly than this seems to have been. I and my team have been developing an asynchronous/non-blocking NIO library for ten years now, and while I'm proud of our work and the discoveries we've made, I am realistic about the fact that it's still pretty far from as good as it could be (not in the least part due to existing platform limitations), certainly far from something I'd say "hey let's standardize this as is". I think that to standardize something of this type that was just written over the past 18-odd months reflects, to put it kindly, some pretty incredible confidence that I wish I shared. Speaking *solely* in the interests of platform quality and integrity, I think that before _any_ high-level non-blocking/asynchronous protocol API is ever introduced into the platform, it would be an incredible waste to not have some kind of design consultation with other industry experts. Now I'm not suggesting that a JDK API would have to be _agreeable_ to every expert, as we all know that is basically impossible; but at the very minimum, I am very confident that we can tell you what _doesn't_ work and the pitfalls we've found along the way, as well as what each of us would consider to be an ideal API, and that is information that has incredible value. Talking about introducing the first-ever non-blocking protocol API into the platform, at _this_ stage, seems premature and needlessly risky. I would suggest that maybe it's best for the API to stick to blocking I/O, at least for now. Or else, take it outside of the platform, and let it mature in the wild where it can evolve without an overbearing concern for compatibility for a decade or so (no, I'm not kidding). As long as this thing lives in the JDK, but isn't standardized, it's probably not going to be used heavily enough to really feel out its weak points. And once it's in, its ability to evolve is severely hampered by compatibility constraints. I feel like it is impossible to over-emphasize the difficulty of the problem of non-blocking I/O when it comes to interactions with user programs. Though the fruits of such an effort are probably small in terms of API surface, the complexity is hard: hard enough that it is, in my mind, a project of a larger scale, maybe JSR scale. And the benefit is potentially large: large enough that it could change the landscape of other specifications and network applications. Or at least, I think so, which is why I've spent so many years of my life in pursuit of such a thing. -- - DML From norman.maurer at googlemail.com Mon Dec 4 18:47:09 2017 From: norman.maurer at googlemail.com (Norman Maurer) Date: Mon, 4 Dec 2017 19:47:09 +0100 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: <07033FCC-0C44-4635-AB3C-002943D7FA76@googlemail.com> Well put David, I couldn?t agree more .... I would even go this far and say this is not something I would include in the platform itself at all. Bye Norman > Am 04.12.2017 um 19:41 schrieb David Lloyd : > >> On Mon, Dec 4, 2017 at 10:17 AM, wrote: >> New JEP Candidate: http://openjdk.java.net/jeps/321 > > I have concerns. > > This will be the first public, NIO-based, asynchronous/non-blocking > network protocol API introduced into the JDK proper, _ever_. > > First, I want to note that the API seems to bear no resemblance > whatsoever with the asynchronous NIO.2 API. Now I'm no fan of that > API as it stands for a couple of reasons, but nevertheless it calls > into question either the validity of the HTTP API as it stands, or the > scope of reusability of the NIO.2 API. > > With that items put aside: there are a wide variety of mature, > non-blocking network protocol implementations out there, with > literally thousands of years of experience distributed amongst their > authors, maintainers, supporters, and communities, none of which were > used as a model. There was, as far as I can see, no kind of study of > existing non-blocking approaches in Java, and their strengths and > weaknesses; there was no round table of engineers with experience in > this field, talking about what works and what doesn't work. > > Some new, seemingly general-purpose, concepts are introduced by the > code base, for example: ByteBufferReference, and ByteBufferPool. Are > these strategies that should be promoted to NIO proper? If not, then > are they _really_ right for this particular use case, particularly if, > for some reason, a _second_ non-blocking network protocol API might be > introduced some day, probably duplicating these concepts? > > Making this thing be the first real platform NIO-based asynchronous > network protocol API, yet being completely removed from the previous > NIO.2 asynchronous APIs and any other existing stable, mature API, > should be done very carefully and deliberately, and perhaps most > importantly, incrementally: first, establish a non-blocking byte > stream API that makes sense generally, and bring that into NIO > (NIO.3?); then, perhaps, enhancements to byte buffers to better > support efficient pooling. By the time that point is reached, it is > hopefully rapidly becoming obvious that this is not something that > should be taken lightly. > > I believe that most third-party implementations are taken less lightly > than this seems to have been. I and my team have been developing an > asynchronous/non-blocking NIO library for ten years now, and while I'm > proud of our work and the discoveries we've made, I am realistic about > the fact that it's still pretty far from as good as it could be (not > in the least part due to existing platform limitations), certainly far > from something I'd say "hey let's standardize this as is". I think > that to standardize something of this type that was just written over > the past 18-odd months reflects, to put it kindly, some pretty > incredible confidence that I wish I shared. > > Speaking *solely* in the interests of platform quality and integrity, > I think that before _any_ high-level non-blocking/asynchronous > protocol API is ever introduced into the platform, it would be an > incredible waste to not have some kind of design consultation with > other industry experts. Now I'm not suggesting that a JDK API would > have to be _agreeable_ to every expert, as we all know that is > basically impossible; but at the very minimum, I am very confident > that we can tell you what _doesn't_ work and the pitfalls we've found > along the way, as well as what each of us would consider to be an > ideal API, and that is information that has incredible value. > > Talking about introducing the first-ever non-blocking protocol API > into the platform, at _this_ stage, seems premature and needlessly > risky. I would suggest that maybe it's best for the API to stick to > blocking I/O, at least for now. Or else, take it outside of the > platform, and let it mature in the wild where it can evolve without an > overbearing concern for compatibility for a decade or so (no, I'm not > kidding). As long as this thing lives in the JDK, but isn't > standardized, it's probably not going to be used heavily enough to > really feel out its weak points. And once it's in, its ability to > evolve is severely hampered by compatibility constraints. > > I feel like it is impossible to over-emphasize the difficulty of the > problem of non-blocking I/O when it comes to interactions with user > programs. Though the fruits of such an effort are probably small in > terms of API surface, the complexity is hard: hard enough that it is, > in my mind, a project of a larger scale, maybe JSR scale. And the > benefit is potentially large: large enough that it could change the > landscape of other specifications and network applications. Or at > least, I think so, which is why I've spent so many years of my life in > pursuit of such a thing. > > -- > - DML From Alan.Bateman at oracle.com Mon Dec 4 20:01:39 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 4 Dec 2017 20:01:39 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: On 04/12/2017 18:41, David Lloyd wrote: > : > Speaking *solely* in the interests of platform quality and integrity, > I think that before _any_ high-level non-blocking/asynchronous > protocol API is ever introduced into the platform, it would be an > incredible waste to not have some kind of design consultation with > other industry experts. Now I'm not suggesting that a JDK API would > have to be _agreeable_ to every expert, as we all know that is > basically impossible; but at the very minimum, I am very confident > that we can tell you what _doesn't_ work and the pitfalls we've found > along the way, as well as what each of us would consider to be an > ideal API, and that is information that has incredible value. > The HTTP client API has been an ongoing effort for several years, the original JEP goes back to 2014. It was initially available in the OpenJDK sandbox and in the JDK main-line before moving to an incubating module in 2016. I can't tell if you've been following this project or not but there has been lots of opportunity to try it out and provide feedback on both the API and implementation. You mention general-purpose concepts such as ByteBufferReference and ByteBufferPool. Note that these are tiny implementation classes (150 lines in total) and not exposed in the API. Promoting these to java.nio would be outside the scope of this API. If the buffer API were to add such concepts in the future then there is nothing to stop the HTTP client implementation making use in its implementation. -Alan From david.lloyd at redhat.com Mon Dec 4 21:56:29 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 4 Dec 2017 15:56:29 -0600 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: On Mon, Dec 4, 2017 at 2:01 PM, Alan Bateman wrote: > On 04/12/2017 18:41, David Lloyd wrote: >> >> : >> Speaking *solely* in the interests of platform quality and integrity, >> I think that before _any_ high-level non-blocking/asynchronous >> protocol API is ever introduced into the platform, it would be an >> incredible waste to not have some kind of design consultation with >> other industry experts. Now I'm not suggesting that a JDK API would >> have to be _agreeable_ to every expert, as we all know that is >> basically impossible; but at the very minimum, I am very confident >> that we can tell you what _doesn't_ work and the pitfalls we've found >> along the way, as well as what each of us would consider to be an >> ideal API, and that is information that has incredible value. >> > The HTTP client API has been an ongoing effort for several years, the > original JEP goes back to 2014. It was initially available in the OpenJDK > sandbox and in the JDK main-line before moving to an incubating module in > 2016. I can't tell if you've been following this project or not but there > has been lots of opportunity to try it out and provide feedback on both the > API and implementation. I've had opportunity to give feedback, perhaps, though the API always seemed incomplete. At least nobody (that I saw) sent out a message saying "Here it is, it's all done, what do you think?". I've certainly never had opportunity to try it out: given its status as an incubating module present only in OpenJDK, the only people who are really in a position to try it out are those using OpenJDK (as opposed to other JDKs) with the flexibility to rewrite their use case if and when the API changes status (being integrated or disappearing) or form (evolving, presumably as a response to feedback), or people writing throwaway applications for the sole purpose of testing this particular API. But those who are best able to make this kind of determination are those who need to be able to immediately use the API, and rely upon it indefinitely (for good or bad), which is definitely not the case for anything incubated in the OpenJDK project. Why take the risk when you can use the Apache HTTP client instead? The lack of feedback on a proposed standard should not be considered a tacit endorsement of it - quite the opposite in fact. It should be considered overwhelming disinterest (i.e. there's nothing particularly compelling about it), or at absolute best, insufficient information to make a determination. The burden should be on the proposer to evangelize their idea and seek out feedback, rather than waiting for interest to appear and feedback to come to them. > You mention general-purpose concepts such as ByteBufferReference and > ByteBufferPool. Note that these are tiny implementation classes (150 lines > in total) and not exposed in the API. Yes they are, currently - at least ByteBufferReference is at the heart of it: http://hg.openjdk.java.net/jdk/jdk/file/6dcbdc9f99fc/src/jdk.incubator.httpclient/share/classes/jdk/incubator/http/AsyncConnection.java#l61 And that class relies directly on ByteBufferPool in its own API. Unless the jdk/jdk branch does not reflect the latest incarnation of this JEP, in which case I definitely haven't have been up to date with it, despite following this list. > Promoting these to java.nio would be > outside the scope of this API. If the buffer API were to add such concepts > in the future then there is nothing to stop the HTTP client implementation > making use in its implementation. See above. -- - DML From david.lloyd at redhat.com Mon Dec 4 22:03:23 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 4 Dec 2017 16:03:23 -0600 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: On Mon, Dec 4, 2017 at 3:56 PM, David Lloyd wrote: > On Mon, Dec 4, 2017 at 2:01 PM, Alan Bateman wrote: >> On 04/12/2017 18:41, David Lloyd wrote: >>> >>> : >>> Speaking *solely* in the interests of platform quality and integrity, >>> I think that before _any_ high-level non-blocking/asynchronous >>> protocol API is ever introduced into the platform, it would be an >>> incredible waste to not have some kind of design consultation with >>> other industry experts. Now I'm not suggesting that a JDK API would >>> have to be _agreeable_ to every expert, as we all know that is >>> basically impossible; but at the very minimum, I am very confident >>> that we can tell you what _doesn't_ work and the pitfalls we've found >>> along the way, as well as what each of us would consider to be an >>> ideal API, and that is information that has incredible value. >>> >> The HTTP client API has been an ongoing effort for several years, the >> original JEP goes back to 2014. It was initially available in the OpenJDK >> sandbox and in the JDK main-line before moving to an incubating module in >> 2016. I can't tell if you've been following this project or not but there >> has been lots of opportunity to try it out and provide feedback on both the >> API and implementation. > > I've had opportunity to give feedback, perhaps, though the API always > seemed incomplete. At least nobody (that I saw) sent out a message > saying "Here it is, it's all done, what do you think?". I've > certainly never had opportunity to try it out: given its status as an > incubating module present only in OpenJDK, the only people who are > really in a position to try it out are those using OpenJDK (as opposed > to other JDKs) with the flexibility to rewrite their use case if and > when the API changes status (being integrated or disappearing) or form > (evolving, presumably as a response to feedback), or people writing > throwaway applications for the sole purpose of testing this particular > API. But those who are best able to make this kind of determination > are those who need to be able to immediately use the API, and rely > upon it indefinitely (for good or bad), which is definitely not the > case for anything incubated in the OpenJDK project. Why take the risk > when you can use the Apache HTTP client instead? > > The lack of feedback on a proposed standard should not be considered a > tacit endorsement of it - quite the opposite in fact. It should be > considered overwhelming disinterest (i.e. there's nothing particularly > compelling about it), or at absolute best, insufficient information to > make a determination. The burden should be on the proposer to > evangelize their idea and seek out feedback, rather than waiting for > interest to appear and feedback to come to them. > >> You mention general-purpose concepts such as ByteBufferReference and >> ByteBufferPool. Note that these are tiny implementation classes (150 lines >> in total) and not exposed in the API. > > Yes they are, currently - at least ByteBufferReference is at the heart of it: > > http://hg.openjdk.java.net/jdk/jdk/file/6dcbdc9f99fc/src/jdk.incubator.httpclient/share/classes/jdk/incubator/http/AsyncConnection.java#l61 I see my error, this is a non-public interface. Nonetheless, I'm not sure that it's really safe to say that this is ready. Has there been _any_ external feedback on this API? -- - DML From chris.hegarty at oracle.com Mon Dec 4 22:04:53 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 4 Dec 2017 22:04:53 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: > On 4 Dec 2017, at 21:56, David Lloyd wrote: > ... >> You mention general-purpose concepts such as ByteBufferReference and >> ByteBufferPool. Note that these are tiny implementation classes (150 lines >> in total) and not exposed in the API. > > Yes they are, currently - at least ByteBufferReference is at the heart of it: > > http://hg.openjdk.java.net/jdk/jdk/file/6dcbdc9f99fc/src/jdk.incubator.httpclient/share/classes/jdk/incubator/http/AsyncConnection.java#l61 > > And that class relies directly on ByteBufferPool in its own API. > > Unless the jdk/jdk branch does not reflect the latest incarnation of > this JEP, in which case I definitely haven't have been up to date with > it, despite following this list. Sorry, before replying further, there is clearly some confusion here. As Alan already said, the above types are NOT part of the API, they are non-public implementation. The ?http-client-branch? of the JDK sandbox [1] contains the very latest code. -Chris. [1] http://hg.openjdk.java.net/jdk/sandbox/branches -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Mon Dec 4 23:01:11 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 4 Dec 2017 23:01:11 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: > On 4 Dec 2017, at 22:03, David Lloyd wrote: >> ... >> >>> You mention general-purpose concepts such as ByteBufferReference and >>> ByteBufferPool. Note that these are tiny implementation classes (150 lines >>> in total) and not exposed in the API. >> >> Yes they are, currently - at least ByteBufferReference is at the heart of it: >> >> http://hg.openjdk.java.net/jdk/jdk/file/6dcbdc9f99fc/src/jdk.incubator.httpclient/share/classes/jdk/incubator/http/AsyncConnection.java#l61 > > I see my error, this is a non-public interface. Nonetheless, I'm not > sure that it's really safe to say that this is ready. Has there been > _any_ external feedback on this API? [ You can probably ignore my previous email, I sent it before receiving your reply. The confusion seems to have been resolved now. ] Yes, there has been external feedback on the API. I?ll dig it up, I need to trawl the net-dev archives and JIRA issues. For your, and others, reference, here is a snapshot of the latest API, built from the ?http-client-branch? of the sandbox: http://cr.openjdk.java.net/~chegar/httpclient/javadoc/api/jdk/incubator/http/package-summary.html -Chris. -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Mon Dec 4 23:09:08 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Mon, 4 Dec 2017 17:09:08 -0600 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: On Mon, Dec 4, 2017 at 5:01 PM, Chris Hegarty wrote: > On 4 Dec 2017, at 22:03, David Lloyd wrote: > > ... > > You mention general-purpose concepts such as ByteBufferReference and > ByteBufferPool. Note that these are tiny implementation classes (150 lines > in total) and not exposed in the API. > > > Yes they are, currently - at least ByteBufferReference is at the heart of > it: > > http://hg.openjdk.java.net/jdk/jdk/file/6dcbdc9f99fc/src/jdk.incubator.httpclient/share/classes/jdk/incubator/http/AsyncConnection.java#l61 > > > I see my error, this is a non-public interface. Nonetheless, I'm not > sure that it's really safe to say that this is ready. Has there been > _any_ external feedback on this API? > > > [ You can probably ignore my previous email, I sent it before receiving your > reply. The confusion seems to have been resolved now. ] > > Yes, there has been external feedback on the API. I?ll dig it up, I need to > trawl the net-dev archives and JIRA issues. > > For your, and others, reference, here is a snapshot of the latest API, built > from the ?http-client-branch? of the sandbox: > > > http://cr.openjdk.java.net/~chegar/httpclient/javadoc/api/jdk/incubator/http/package-summary.html Thanks. If you don't mind answering one more question: is there any possibility to intercept authentication completely? There does not seem to be a lot of documentation about authentication in this API, or what happens (for example) if there is no Authenticator. -- - DML From martinrb at google.com Tue Dec 5 04:01:09 2017 From: martinrb at google.com (Martin Buchholz) Date: Mon, 4 Dec 2017 20:01:09 -0800 Subject: RFR: 8193034: Optimize URL.toExternalForm Message-ID: http://cr.openjdk.java.net/~martin/webrevs/jdk/URL.toExternalForm/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph.langer at sap.com Tue Dec 5 07:50:03 2017 From: christoph.langer at sap.com (Langer, Christoph) Date: Tue, 5 Dec 2017 07:50:03 +0000 Subject: 8193034: Optimize URL.toExternalForm In-Reply-To: References: Message-ID: Hi Martin, I think the new code is more compact and readable which makes it nice. I reviewed it to check that the result is still the same as before. However, I can?t assess if it is acceptable from a performance point of view and would defer this assessment to some other reviewer. But apart from performance constraints the change seems good. Best regards Christoph From: net-dev [mailto:net-dev-bounces at openjdk.java.net] On Behalf Of Martin Buchholz Sent: Dienstag, 5. Dezember 2017 05:01 To: net-dev Subject: RFR: 8193034: Optimize URL.toExternalForm http://cr.openjdk.java.net/~martin/webrevs/jdk/URL.toExternalForm/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Dec 5 09:10:00 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 5 Dec 2017 09:10:00 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: <9fb7d98f-62b5-46fd-0e7f-76e3da12f7bb@oracle.com> On 04/12/2017 21:56, David Lloyd wrote: > : > I've had opportunity to give feedback, perhaps, though the API always > seemed incomplete. At least nobody (that I saw) sent out a message > saying "Here it is, it's all done, what do you think?". I've > certainly never had opportunity to try it out: given its status as an > incubating module present only in OpenJDK, the only people who are > really in a position to try it out are those using OpenJDK (as opposed > to other JDKs) with the flexibility to rewrite their use case if and > when the API changes status (being integrated or disappearing) or form > (evolving, presumably as a response to feedback), or people writing > throwaway applications for the sole purpose of testing this particular > API. But those who are best able to make this kind of determination > are those who need to be able to immediately use the API, and rely > upon it indefinitely (for good or bad), which is definitely not the > case for anything incubated in the OpenJDK project. Incubator modules (JEP 11) is the means to get non-final APIs and features into the hands of developers. In this case, the HTTP client was an incubator module in JDK 9 and is proposed to continue (with a new implementation and some API refinements [1]) as an incubator module in JDK 10. So there has been lots of time to try out the API, send feedback, contribute, etc. Yes, the onus is on interested developers to get involved and there has been useful feedback on this mailing mail. If you read JEP 11 then you'll know that an API or feature can't stay in the incubator forever, it either moves forever or it is removed. In this case, the API has been through numerous iterations and is looking quite good, hence JEP 321 proposes to move it forward so that it can be promoted to a platform module. -Alan. [1] http://mail.openjdk.java.net/pipermail/net-dev/2017-November/011017.html From chris.hegarty at oracle.com Tue Dec 5 10:10:06 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 5 Dec 2017 10:10:06 +0000 Subject: RFR: 8193034: Optimize URL.toExternalForm In-Reply-To: References: Message-ID: <8B8F59AA-A5B3-4173-9B1F-7A726D8C609D@oracle.com> > On 5 Dec 2017, at 04:01, Martin Buchholz wrote: > > http://cr.openjdk.java.net/~martin/webrevs/jdk/URL.toExternalForm/ Since the motivation for this change is performance, do you have any performance numbers / benchmarks that you can share? -Chris. From chris.hegarty at oracle.com Tue Dec 5 10:49:44 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 5 Dec 2017 10:49:44 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: <966178F4-7901-40DA-BA52-C66F3BF162B2@oracle.com> > On 4 Dec 2017, at 23:09, David Lloyd wrote: >> ... > > Thanks. If you don't mind answering one more question: is there any > possibility to intercept authentication completely? As with HttpURLConnection, if no Authenticator is set then, application code should be able to manually inspect and set the appropriate headers to perform authentication. -Chris. From Alan.Bateman at oracle.com Tue Dec 5 10:55:14 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 5 Dec 2017 10:55:14 +0000 Subject: RFR: 8193034: Optimize URL.toExternalForm In-Reply-To: References: Message-ID: On 05/12/2017 04:01, Martin Buchholz wrote: > http://cr.openjdk.java.net/~martin/webrevs/jdk/URL.toExternalForm/ > The style is interesting but I don't see anything obvious wrong. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalibor.topic at oracle.com Tue Dec 5 11:48:44 2017 From: dalibor.topic at oracle.com (dalibor topic) Date: Tue, 5 Dec 2017 12:48:44 +0100 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: <88141ea6-f5f0-f8e5-8c0c-d32b26e808cd@oracle.com> On 04.12.2017 22:56, David Lloyd wrote: > saying "Here it is, it's all done, what do you think?". I've > certainly never had opportunity to try it out: given its status as an > incubating module present only in OpenJDK, the only people who are > really in a position to try it out are those using OpenJDK (as opposed > to other JDKs) That's not quite correct. The jdk.incubator.httpclient module is part of Oracle JDK 9, as well. It has been part of the JDK 9 Early Access builds since about a year ago, afaik. The Early Access builds can be found at jdk.java.net, fwiw. > But those who are best able to make this kind of determination > are those who need to be able to immediately use the API, and rely > upon it indefinitely (for good or bad), which is definitely not the > case for anything incubated in the OpenJDK project. This seems to be a more general criticism of the incubator module mechanism as defined in JEP 11, rather than directed at this API specifically. It wasn't raised on jdk-dev when JEP 11 was discussed about a year ago, fwiw. I would take issue with the qualifier 'best' in the above, as the qualification offered for making best determinations is necessity, rather than expertise, for example. In some cases, both properties may hold, of course, but I don't think that's a valid assertion in general. cheers, dalibor topic -- Dalibor Topic | Principal Product Manager Phone: +494089091214 | Mobile: +491737185961 ORACLE Deutschland B.V. & Co. KG | K?hneh?fe 5 | 22761 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 M?nchen Registergericht: Amtsgericht M?nchen, HRA 95603 Komplement?rin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Gesch?ftsf?hrer: Alexander van der Ven, Jan Schultheiss, Val Maher Oracle is committed to developing practices and products that help protect the environment From david.lloyd at redhat.com Tue Dec 5 13:48:38 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Tue, 5 Dec 2017 07:48:38 -0600 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: <88141ea6-f5f0-f8e5-8c0c-d32b26e808cd@oracle.com> References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> <88141ea6-f5f0-f8e5-8c0c-d32b26e808cd@oracle.com> Message-ID: On Tue, Dec 5, 2017 at 5:48 AM, dalibor topic wrote: > On 04.12.2017 22:56, David Lloyd wrote: >> >> saying "Here it is, it's all done, what do you think?". I've >> certainly never had opportunity to try it out: given its status as an >> incubating module present only in OpenJDK, the only people who are >> really in a position to try it out are those using OpenJDK (as opposed >> to other JDKs) > > That's not quite correct. The jdk.incubator.httpclient module is part of > Oracle JDK 9, as well. It has been part of the JDK 9 Early Access builds > since about a year ago, afaik. The Early Access builds can be found at > jdk.java.net, fwiw. Sure, but that's all the same ecosystem. An organization tied to, say, the IBM JDK won't have access to these. >> But those who are best able to make this kind of determination >> are those who need to be able to immediately use the API, and rely >> upon it indefinitely (for good or bad), which is definitely not the >> case for anything incubated in the OpenJDK project. > > This seems to be a more general criticism of the incubator module mechanism > as defined in JEP 11, rather than directed at this API specifically. It > wasn't raised on jdk-dev when JEP 11 was discussed about a year ago, fwiw. Of course; sometimes a thing must be tried before the weaknesses are apparent. If we all designed perfect things from the start, we wouldn't need standards and evolution. :) -- - DML From dalibor.topic at oracle.com Tue Dec 5 14:05:39 2017 From: dalibor.topic at oracle.com (dalibor topic) Date: Tue, 5 Dec 2017 15:05:39 +0100 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> <88141ea6-f5f0-f8e5-8c0c-d32b26e808cd@oracle.com> Message-ID: On 05.12.2017 14:48, David Lloyd wrote: > On Tue, Dec 5, 2017 at 5:48 AM, dalibor topic wrote: >> On 04.12.2017 22:56, David Lloyd wrote: >>> >>> saying "Here it is, it's all done, what do you think?". I've >>> certainly never had opportunity to try it out: given its status as an >>> incubating module present only in OpenJDK, the only people who are >>> really in a position to try it out are those using OpenJDK (as opposed >>> to other JDKs) >> >> That's not quite correct. The jdk.incubator.httpclient module is part of >> Oracle JDK 9, as well. It has been part of the JDK 9 Early Access builds >> since about a year ago, afaik. The Early Access builds can be found at >> jdk.java.net, fwiw. > > Sure, but that's all the same ecosystem. An organization tied to, > say, the IBM JDK won't have access to these. Fwiw, IBM's OpenJ9 builds for JDK 9 would seem to include this incubator module as well. I'd naively assume the same to be the case for Azul's JDK 9 builds, etc. >> This seems to be a more general criticism of the incubator module mechanism >> as defined in JEP 11, rather than directed at this API specifically. It >> wasn't raised on jdk-dev when JEP 11 was discussed about a year ago, fwiw. > > Of course; sometimes a thing must be tried before the weaknesses are > apparent. If we all designed perfect things from the start, we > wouldn't need standards and evolution. :) Touch? - The sample size of one for modules going through the incubation module mechanism may still be too small for a categorical critique, though. ;) cheers, dalibor topic -- Dalibor Topic | Principal Product Manager Phone: +494089091214 | Mobile: +491737185961 ORACLE Deutschland B.V. & Co. KG | K?hneh?fe 5 | 22761 Hamburg ORACLE Deutschland B.V. & Co. KG Hauptverwaltung: Riesstr. 25, D-80992 M?nchen Registergericht: Amtsgericht M?nchen, HRA 95603 Komplement?rin: ORACLE Deutschland Verwaltung B.V. Hertogswetering 163/167, 3543 AS Utrecht, Niederlande Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697 Gesch?ftsf?hrer: Alexander van der Ven, Jan Schultheiss, Val Maher Oracle is committed to developing practices and products that help protect the environment From viktor.klang at gmail.com Wed Dec 6 10:01:29 2017 From: viktor.klang at gmail.com (Viktor Klang) Date: Wed, 6 Dec 2017 11:01:29 +0100 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: Hi David, I think you raise some valid concerns, particularly with the byte buffer pools. That said, I'm interested in knowing whether your concerns extend to choice of using the JDK9 Flow API for handling asynchronous IO? Cheers, ? (Adding my colleague, James Roper, to the conversation) On Dec 4, 2017 19:42, "David Lloyd" wrote: > On Mon, Dec 4, 2017 at 10:17 AM, wrote: > > New JEP Candidate: http://openjdk.java.net/jeps/321 > > I have concerns. > > This will be the first public, NIO-based, asynchronous/non-blocking > network protocol API introduced into the JDK proper, _ever_. > > First, I want to note that the API seems to bear no resemblance > whatsoever with the asynchronous NIO.2 API. Now I'm no fan of that > API as it stands for a couple of reasons, but nevertheless it calls > into question either the validity of the HTTP API as it stands, or the > scope of reusability of the NIO.2 API. > > With that items put aside: there are a wide variety of mature, > non-blocking network protocol implementations out there, with > literally thousands of years of experience distributed amongst their > authors, maintainers, supporters, and communities, none of which were > used as a model. There was, as far as I can see, no kind of study of > existing non-blocking approaches in Java, and their strengths and > weaknesses; there was no round table of engineers with experience in > this field, talking about what works and what doesn't work. > > Some new, seemingly general-purpose, concepts are introduced by the > code base, for example: ByteBufferReference, and ByteBufferPool. Are > these strategies that should be promoted to NIO proper? If not, then > are they _really_ right for this particular use case, particularly if, > for some reason, a _second_ non-blocking network protocol API might be > introduced some day, probably duplicating these concepts? > > Making this thing be the first real platform NIO-based asynchronous > network protocol API, yet being completely removed from the previous > NIO.2 asynchronous APIs and any other existing stable, mature API, > should be done very carefully and deliberately, and perhaps most > importantly, incrementally: first, establish a non-blocking byte > stream API that makes sense generally, and bring that into NIO > (NIO.3?); then, perhaps, enhancements to byte buffers to better > support efficient pooling. By the time that point is reached, it is > hopefully rapidly becoming obvious that this is not something that > should be taken lightly. > > I believe that most third-party implementations are taken less lightly > than this seems to have been. I and my team have been developing an > asynchronous/non-blocking NIO library for ten years now, and while I'm > proud of our work and the discoveries we've made, I am realistic about > the fact that it's still pretty far from as good as it could be (not > in the least part due to existing platform limitations), certainly far > from something I'd say "hey let's standardize this as is". I think > that to standardize something of this type that was just written over > the past 18-odd months reflects, to put it kindly, some pretty > incredible confidence that I wish I shared. > > Speaking *solely* in the interests of platform quality and integrity, > I think that before _any_ high-level non-blocking/asynchronous > protocol API is ever introduced into the platform, it would be an > incredible waste to not have some kind of design consultation with > other industry experts. Now I'm not suggesting that a JDK API would > have to be _agreeable_ to every expert, as we all know that is > basically impossible; but at the very minimum, I am very confident > that we can tell you what _doesn't_ work and the pitfalls we've found > along the way, as well as what each of us would consider to be an > ideal API, and that is information that has incredible value. > > Talking about introducing the first-ever non-blocking protocol API > into the platform, at _this_ stage, seems premature and needlessly > risky. I would suggest that maybe it's best for the API to stick to > blocking I/O, at least for now. Or else, take it outside of the > platform, and let it mature in the wild where it can evolve without an > overbearing concern for compatibility for a decade or so (no, I'm not > kidding). As long as this thing lives in the JDK, but isn't > standardized, it's probably not going to be used heavily enough to > really feel out its weak points. And once it's in, its ability to > evolve is severely hampered by compatibility constraints. > > I feel like it is impossible to over-emphasize the difficulty of the > problem of non-blocking I/O when it comes to interactions with user > programs. Though the fruits of such an effort are probably small in > terms of API surface, the complexity is hard: hard enough that it is, > in my mind, a project of a larger scale, maybe JSR scale. And the > benefit is potentially large: large enough that it could change the > landscape of other specifications and network applications. Or at > least, I think so, which is why I've spent so many years of my life in > pursuit of such a thing. > > -- > - DML > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Wed Dec 6 12:31:11 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 6 Dec 2017 12:31:11 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: Viktor, I would like to address your first comment only, as your question is directed to someone else. > On 6 Dec 2017, at 10:01, Viktor Klang wrote: > .. > I think you raise some valid concerns, particularly with the byte buffer pools. The conversation got off to a bad start as there was a misunderstanding about what was actually being proposed, so I would like to clear that up. The reference to byte buffer pools is not relevant here. They were introduced as a performance optimization that allowed better reuse of byte buffers between the socket channel and the SSL engine, in the implementation. That's all, nothing more. They have no bearing on the API, and this JEP is not proposing to make any changes in the NIO area. What is relevant is the use of byte buffers as a Flow of request and response body data. The API provides no mechanism for reuse, or pooling, of these byte buffers ( it did in a previous revision but was removed because of its complexity ). The latest version of the API for BodySubscriber [1] contains the following: ?... the ByteBuffers, once passed to the subscriber, are no longer used by the HTTP client.?, and the BodyPublisher [2] contains: ?Instances of ByteBuffer published by the publisher must be allocated by the publisher, and must not be accessed after being handed over to the library.". I accept that this could be made more clear in the API. The API uses byte buffers as carriers of data in a way that is consistent with other APIs. In the case of the publisher, it is the responsibility of the user to allocate the buffer and pass it to the HTTP Client, after which it should not be modified by user code. In the case of the subscriber, once the byte buffer is passed to `onNext`, it is then the sole responsibility of the subscriber. The primary motivation for the use byte buffers, as described above, is to provide maximum flexibility to an implementation to avoid copying and buffering of data. --- Through contacts from Viktor, we have recently received some feedback on the use of the body subscriber and publisher in the API. This feedback is very helpful and we will soon send out a proposal that will better support integration with existing publishers and subscribers. [ I will work with Viktor to bring this to the mailing list ]. -Chris. [1] http://cr.openjdk.java.net/~chegar/httpclient/javadoc/api/jdk/incubator/http/HttpResponse.BodySubscriber.html [2] http://cr.openjdk.java.net/~chegar/httpclient/javadoc/api/jdk/incubator/http/HttpRequest.BodyPublisher.html From viktor.klang at gmail.com Wed Dec 6 12:34:42 2017 From: viktor.klang at gmail.com (Viktor Klang) Date: Wed, 6 Dec 2017 13:34:42 +0100 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: Thanks for those clarifcations, Chris, I really appreciate it! -- Cheers, ? On Dec 6, 2017 13:31, "Chris Hegarty" wrote: > Viktor, > > I would like to address your first comment only, as your question is > directed to someone else. > > > On 6 Dec 2017, at 10:01, Viktor Klang wrote: > > .. > > I think you raise some valid concerns, particularly with the byte buffer > pools. > > The conversation got off to a bad start as there was a misunderstanding > about what was actually being proposed, so I would like to clear that > up. > > The reference to byte buffer pools is not relevant here. They were > introduced as a performance optimization that allowed better reuse of > byte buffers between the socket channel and the SSL engine, in the > implementation. That's all, nothing more. They have no bearing on the > API, and this JEP is not proposing to make any changes in the NIO area. > > What is relevant is the use of byte buffers as a Flow of request and > response body data. The API provides no mechanism for reuse, or pooling, > of these byte buffers ( it did in a previous revision but was removed > because of its complexity ). The latest version of the API for > BodySubscriber [1] contains the following: ?... the ByteBuffers, > once passed to the subscriber, are no longer used by the HTTP client.?, > and the BodyPublisher [2] contains: ?Instances of ByteBuffer published > by the publisher must be allocated by the publisher, and must not be > accessed after being handed over to the library.". I accept that this > could be made more clear in the API. > > The API uses byte buffers as carriers of data in a way that is > consistent with other APIs. In the case of the publisher, it is the > responsibility of the user to allocate the buffer and pass it to the > HTTP Client, after which it should not be modified by user code. In the > case of the subscriber, once the byte buffer is passed to `onNext`, it > is then the sole responsibility of the subscriber. > > The primary motivation for the use byte buffers, as described above, is > to provide maximum flexibility to an implementation to avoid copying > and buffering of data. > > --- > > Through contacts from Viktor, we have recently received some feedback on > the use of the body subscriber and publisher in the API. This feedback > is very helpful and we will soon send out a proposal that will better > support integration with existing publishers and subscribers. [ I will > work with Viktor to bring this to the mailing list ]. > > > -Chris. > > [1] http://cr.openjdk.java.net/~chegar/httpclient/javadoc/api/ > jdk/incubator/http/HttpResponse.BodySubscriber.html > [2] http://cr.openjdk.java.net/~chegar/httpclient/javadoc/api/ > jdk/incubator/http/HttpRequest.BodyPublisher.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Wed Dec 6 19:58:37 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Wed, 6 Dec 2017 13:58:37 -0600 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: On Wed, Dec 6, 2017 at 6:31 AM, Chris Hegarty wrote: > Viktor, > > The reference to byte buffer pools is not relevant here. They were > introduced as a performance optimization that allowed better reuse of > byte buffers between the socket channel and the SSL engine, in the > implementation. That's all, nothing more. They have no bearing on the > API, and this JEP is not proposing to make any changes in the NIO area. +1, that was my fault, sorry about that. > What is relevant is the use of byte buffers as a Flow of request and > response body data. The API provides no mechanism for reuse, or pooling, > of these byte buffers ( it did in a previous revision but was removed > because of its complexity ). The latest version of the API for > BodySubscriber [1] contains the following: ?... the ByteBuffers, > once passed to the subscriber, are no longer used by the HTTP client.?, > and the BodyPublisher [2] contains: ?Instances of ByteBuffer published > by the publisher must be allocated by the publisher, and must not be > accessed after being handed over to the library.". Just out of curiosity, what kind of issues came up? What did the mechanism look like - was it just a callback to return depleted buffers to the HTTP client or user (as appropriate)? It sure would be nice to somehow finally tackle the problem of direct buffer allocation, pooling, and freeing for good; it's a bit of a nightmare right now. The lack of a good, standard pooling strategy is a big part of what makes me nervous about introducing an entire new async paradigm based on buffers, particularly with no possibility for pooling or reuse. I have many thoughts in this area (*crowd groans*) but it would be interesting to know what practical problems were encountered. > The API uses byte buffers as carriers of data in a way that is > consistent with other APIs. In the case of the publisher, it is the > responsibility of the user to allocate the buffer and pass it to the > HTTP Client, after which it should not be modified by user code. In the > case of the subscriber, once the byte buffer is passed to `onNext`, it > is then the sole responsibility of the subscriber. > > The primary motivation for the use byte buffers, as described above, is > to provide maximum flexibility to an implementation to avoid copying > and buffering of data. Is my reading of the API correct in that flow control is happening in terms of buffers, not of bytes? Could there ever be any odd effects from very small or very large buffers passing through the plumbing? -- - DML From james at lightbend.com Thu Dec 7 00:19:12 2017 From: james at lightbend.com (James Roper) Date: Thu, 7 Dec 2017 11:19:12 +1100 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: On 7 December 2017 at 06:58, David Lloyd wrote: > On Wed, Dec 6, 2017 at 6:31 AM, Chris Hegarty > wrote: [snip] > > The primary motivation for the use byte buffers, as described above, is > > to provide maximum flexibility to an implementation to avoid copying > > and buffering of data. > > Is my reading of the API correct in that flow control is happening in > terms of buffers, not of bytes? Could there ever be any odd effects > from very small or very large buffers passing through the plumbing? > Your reading is correct. In my experience, it varies wildly by use case. In the technology I work with (Akka), we do exactly this, we have ByteStrings (essentially immutable byte buffer), and flow control is done on the number of ByteStrings, not the number of bytes in those strings. Generally, in reading, the size of ByteStrings is limited by a configurable amount, for example, 8kb. And then Akka's flow control will, by default, keep up to 8 ByteStrings in flight in its asynchronous processing pipeline. So we have a maximum buffer size of 64kb per connection. For most HTTP use cases, this is fine, something reading an HTTP message body might be collecting those buffers up to a maximum size of 100kb by default, and then parsing the buffer (eg, as json). So it's within the tolerances of what the amount of memory that the user expects to use per request. If the data read in to the buffers were very small, this would be due to the client trickle feeding the server - care must be taken on the server to ensure that if 8kb buffers are allocated for reads, but only a small amount of data is read, that these large buffers are released, and the small data copied to a small buffer. I think where it can possible cause a problem is if for some reason something sending data is only generating small byte buffer chunks, but there's a long (and expensive) pipeline for the chunks to go through before they get written out. This is not a use case that we see that often, but I have seen it. The solution there is to either increase the number of elements in flight in the stream (most reactive streams implementations allow this to be done trivially), or to put an aggregating buffer in the middle before the expensive processing (again, streaming implementations such as RxJava, Reactor or Akka streams provide straight forward stages to do this). One issue that I'm not sure about is the consequences of using direct buffers with regards to garbage collection. If direct buffers are never copied onto the heap, and are never reused, lets say you're just implementing a proxy passing buffers through from one connection to another, then the heap usage of the application may be very small, and this could mean that garbage collection is done very infrequently. As I understand it, this can result in direct buffers staying around for a long time, and possibly causing the system to run out of memory. Does anyone have any experience with that, and how to deal with it? We don't generally have this problem in Akka because we always copy our buffers onto the heap into an immutable structure, so even if we do use direct buffers and don't reuse them, our heap usage grows at least as fast as our direct buffer usage grows, which means total memory usage won't exceed twice the size of the heap since eventually garbage collection will clean both up. > > -- > - DML > -- *James Roper* *Senior Octonaut* Lightbend ? Build reactive apps! Twitter: @jroper -------------- next part -------------- An HTML attachment was scrubbed... URL: From viktor.klang at gmail.com Thu Dec 7 07:01:34 2017 From: viktor.klang at gmail.com (Viktor Klang) Date: Thu, 7 Dec 2017 08:01:34 +0100 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: Sidenote: a byte is an 8-bit buffer chunk. :-) -- Cheers, ? On Dec 7, 2017 01:19, "James Roper" wrote: > On 7 December 2017 at 06:58, David Lloyd wrote: > >> On Wed, Dec 6, 2017 at 6:31 AM, Chris Hegarty >> wrote: > > > [snip] > > >> > The primary motivation for the use byte buffers, as described above, is >> > to provide maximum flexibility to an implementation to avoid copying >> > and buffering of data. >> >> Is my reading of the API correct in that flow control is happening in >> terms of buffers, not of bytes? Could there ever be any odd effects >> from very small or very large buffers passing through the plumbing? >> > > Your reading is correct. In my experience, it varies wildly by use case. > In the technology I work with (Akka), we do exactly this, we have > ByteStrings (essentially immutable byte buffer), and flow control is done > on the number of ByteStrings, not the number of bytes in those strings. > Generally, in reading, the size of ByteStrings is limited by a configurable > amount, for example, 8kb. And then Akka's flow control will, by default, > keep up to 8 ByteStrings in flight in its asynchronous processing pipeline. > So we have a maximum buffer size of 64kb per connection. For most HTTP use > cases, this is fine, something reading an HTTP message body might be > collecting those buffers up to a maximum size of 100kb by default, and then > parsing the buffer (eg, as json). So it's within the tolerances of what the > amount of memory that the user expects to use per request. If the data read > in to the buffers were very small, this would be due to the client trickle > feeding the server - care must be taken on the server to ensure that if 8kb > buffers are allocated for reads, but only a small amount of data is read, > that these large buffers are released, and the small data copied to a small > buffer. > > I think where it can possible cause a problem is if for some reason > something sending data is only generating small byte buffer chunks, but > there's a long (and expensive) pipeline for the chunks to go through before > they get written out. This is not a use case that we see that often, but I > have seen it. The solution there is to either increase the number of > elements in flight in the stream (most reactive streams implementations > allow this to be done trivially), or to put an aggregating buffer in the > middle before the expensive processing (again, streaming implementations > such as RxJava, Reactor or Akka streams provide straight forward stages to > do this). > > One issue that I'm not sure about is the consequences of using direct > buffers with regards to garbage collection. If direct buffers are never > copied onto the heap, and are never reused, lets say you're just > implementing a proxy passing buffers through from one connection to > another, then the heap usage of the application may be very small, and this > could mean that garbage collection is done very infrequently. As I > understand it, this can result in direct buffers staying around for a long > time, and possibly causing the system to run out of memory. Does anyone > have any experience with that, and how to deal with it? We don't generally > have this problem in Akka because we always copy our buffers onto the heap > into an immutable structure, so even if we do use direct buffers and don't > reuse them, our heap usage grows at least as fast as our direct buffer > usage grows, which means total memory usage won't exceed twice the size of > the heap since eventually garbage collection will clean both up. > > >> >> -- >> - DML >> > > > > -- > *James Roper* > *Senior Octonaut* > > Lightbend ? Build reactive apps! > Twitter: @jroper > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Thu Dec 7 11:53:42 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Thu, 7 Dec 2017 11:53:42 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: <29f279d7-1f3e-ec90-268f-aa913b974b82@oracle.com> On 07/12/2017 00:19, James Roper wrote: > : > > One issue that I'm not sure about is the consequences of using direct > buffers with regards to garbage collection. If direct buffers are > never copied onto the heap, and are never reused, lets say you're just > implementing a proxy passing buffers through from one connection to > another, then the heap usage of the application may be very small, and > this could mean that garbage collection is done very infrequently. As > I understand it, this can result in direct buffers staying around for > a long time, and possibly causing the system to run out of memory. > Does anyone have any experience with that, and how to deal with it? We > don't generally have this problem in Akka because we always copy our > buffers onto the heap into an immutable structure, so even if we do > use direct buffers and don't reuse them, our heap usage grows at least > as fast as our direct buffer usage grows, which means total memory > usage won't exceed twice the size of the heap since eventually garbage > collection will clean both up. This thread is getting a little off-topic but just to say that you can limit the amount of memory reserved for direct buffers. Attempts to allocate further direct buffers will trigger GC and reference processing. Yes, this can be expensive. This area improved significantly in JDK 9 so that the retry coordinates with the reference processing and mostly avoids spurious OOME that were possible with older releases. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lloyd at redhat.com Thu Dec 7 13:14:53 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 7 Dec 2017 07:14:53 -0600 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: <29f279d7-1f3e-ec90-268f-aa913b974b82@oracle.com> References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> <29f279d7-1f3e-ec90-268f-aa913b974b82@oracle.com> Message-ID: On Thu, Dec 7, 2017 at 5:53 AM, Alan Bateman wrote: > This thread is getting a little off-topic but... Getting it back on topic again: > Proposal for the standard module name: java.net.httpclient. Proposal for the standard package name: java.net.http. I think it would be better if both the module and the package were "java.net.http.client", for two reasons. Firstly, it is important to align the package and module name whenever possible; I don't think there's a compelling reason here (or most anywhere else) not to do so. Secondly, it is not unreasonable to expect that the future may bring other HTTP-related APIs that are not necessarily client-specific. Relatedly, it may be wise to rename "HttpRequest" and "HttpResponse" to "HttpClientRequest" and "HttpClientResponse", respectively. -- - DML From chris.hegarty at oracle.com Thu Dec 7 14:32:09 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 7 Dec 2017 14:32:09 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> <29f279d7-1f3e-ec90-268f-aa913b974b82@oracle.com> Message-ID: David, On 07/12/17 13:14, David Lloyd wrote: > On Thu, Dec 7, 2017 at 5:53 AM, Alan Bateman wrote: >> This thread is getting a little off-topic but... > > Getting it back on topic again: > >> Proposal for the standard module name: java.net.httpclient. Proposal for the standard package name: java.net.http. > > I think it would be better if both the module and the package were > "java.net.http.client", for two reasons. Firstly, it is important to > align the package and module name whenever possible; I don't think > there's a compelling reason here (or most anywhere else) not to do so. > Secondly, it is not unreasonable to expect that the future may bring > other HTTP-related APIs that are not necessarily client-specific. > > Relatedly, it may be wise to rename "HttpRequest" and "HttpResponse" > to "HttpClientRequest" and "HttpClientResponse", respectively. I agree that symmetry between the module name and package name is desirable. If the module / package name contains `client`, then it is effectively redundant in the type name ( unless there are many types being imported from other unrelated HTTP libraries ). The JEP does make a proposal on the module and package name, but I think a discussion on naming will be needed. I'm not sure that we need to decide that now, but your point has been noted. -Chris. From chris.hegarty at oracle.com Thu Dec 7 15:40:15 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Thu, 7 Dec 2017 15:40:15 +0000 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> Message-ID: James, On 07/12/17 00:19, James Roper wrote: > ... > Your reading is correct. In my experience, it varies wildly by use case. > In the technology I work with (Akka), we do exactly this, we have > ByteStrings (essentially immutable byte buffer), and flow control is > done on the number of ByteStrings, not the number of bytes in those > strings. Generally, in reading, the size of ByteStrings is limited by a > configurable amount, for example, 8kb. And then Akka's flow control > will, by default, keep up to 8 ByteStrings in flight in its asynchronous > processing pipeline. So we have a maximum buffer size of 64kb per > connection. For most HTTP use cases, this is fine, something reading an > HTTP message body might be collecting those buffers up to a maximum size > of 100kb by default, and then parsing the buffer (eg, as json). So it's > within the tolerances of what the amount of memory that the user expects > to use per request. If the data read in to the buffers were very small, > this would be due to the client trickle feeding the server - care must > be taken on the server to ensure that if 8kb buffers are allocated for > reads, but only a small amount of data is read, that these large buffers > are released, and the small data copied to a small buffer. > > I think where it can possible cause a problem is if for some reason > something sending data is only generating small byte buffer chunks, but > there's a long (and expensive) pipeline for the chunks to go through > before they get written out. This is not a use case that we see that > often, but I have seen it. The solution there is to either increase the > number of elements in flight in the stream (most reactive streams > implementations allow this to be done trivially), or to put an > aggregating buffer in the middle before the expensive processing (again, > streaming implementations such as RxJava, Reactor or Akka streams > provide straight forward stages to do this). Part of the feedback [1] we received to date has resulted in the addition of a buffering subscriber [2], that buffers data before delivering it to a downstream subscriber. The buffering subscriber guarantees to deliver a given number of bytes of data to each invocation of the downstream's `onNext`, except for the final invocation, just before `onComplete` is invoked. This lead us to the realization that an aggregate of byte buffers is more flexible, it allows for, but does not require, accumulation where it makes sense. We encountered similar issues with trickling small amounts of data, some of which are described in 8186750 [3]. We settled on a a similar solution, accumulate below a certain threshold. This can be important in cases where the data is framed, like in HTTP/2. > One issue that I'm not sure about is the consequences of using direct > buffers with regards to garbage collection. If direct buffers are never > copied onto the heap, and are never reused, lets say you're just > implementing a proxy passing buffers through from one connection to > another, then the heap usage of the application may be very small, and > this could mean that garbage collection is done very infrequently. As I > understand it, this can result in direct buffers staying around for a > long time, and possibly causing the system to run out of memory. Does > anyone have any experience with that, and how to deal with it? We don't > generally have this problem in Akka because we always copy our buffers > onto the heap into an immutable structure, so even if we do use direct > buffers and don't reuse them, our heap usage grows at least as fast as > our direct buffer usage grows, which means total memory usage won't > exceed twice the size of the heap since eventually garbage collection > will clean both up. We found that performance / throughput is far more dependent on factors other than direct buffers, for example in HTTP/2 the connection / stream window, and frame sizes have a significant impact. Given the prevalence of HTTPS, the data on and off the wire requires encryption and decryption, much of which does not appear to benefit from being in direct buffers. Additionally, buffer management becomes tricky when a single buffer can contain multiple HTTP/2 frames, slicing, tracking, etc, is required to avoiding copying, if the intent is to support reuse of the buffer. Much of our experiments with direct buffers did not yield any obvious benefit, and as such the implementation we have today only uses heap buffers. That said, the API does not preclude the use of direct buffers, or the addition of more advanced buffer management in the future. Alan has already covered some of the recent improvements to freeing direct buffers. -Chris. [1] https://bugs.openjdk.java.net/browse/JDK-8184285 [2] http://cr.openjdk.java.net/~chegar/httpclient/javadoc/api/jdk/incubator/http/HttpResponse.BodySubscriber.html#buffering(jdk.incubator.http.HttpResponse.BodySubscriber,int) [3] https://bugs.openjdk.java.net/browse/JDK-8186750 From david.lloyd at redhat.com Thu Dec 7 23:32:29 2017 From: david.lloyd at redhat.com (David Lloyd) Date: Thu, 7 Dec 2017 17:32:29 -0600 Subject: JEP 321: HTTP Client (Standard) In-Reply-To: References: <20171204161709.166BE10C4B1@eggemoggin.niobe.net> <29f279d7-1f3e-ec90-268f-aa913b974b82@oracle.com> Message-ID: On Thu, Dec 7, 2017 at 8:32 AM, Chris Hegarty wrote: > David, > > On 07/12/17 13:14, David Lloyd wrote: >> >> On Thu, Dec 7, 2017 at 5:53 AM, Alan Bateman >> wrote: >>> >>> This thread is getting a little off-topic but... >> >> >> Getting it back on topic again: >> >>> Proposal for the standard module name: java.net.httpclient. Proposal for >>> the standard package name: java.net.http. >> >> >> I think it would be better if both the module and the package were >> "java.net.http.client", for two reasons. Firstly, it is important to >> align the package and module name whenever possible; I don't think >> there's a compelling reason here (or most anywhere else) not to do so. >> Secondly, it is not unreasonable to expect that the future may bring >> other HTTP-related APIs that are not necessarily client-specific. >> >> Relatedly, it may be wise to rename "HttpRequest" and "HttpResponse" >> to "HttpClientRequest" and "HttpClientResponse", respectively. > > > I agree that symmetry between the module name and package name is > desirable. If the module / package name contains `client`, then it is > effectively redundant in the type name ( unless there are many types > being imported from other unrelated HTTP libraries ). HttpRequest and HttpResponse are very common names for both client and server libraries. I would not consider it odd for a server to have HttpRequest for incoming requests; if the client also uses HttpRequest for its outbound requests, then a bunch of annoying qualification will be going on. It seems redundant, but the namespacing of the package and the class serve different purposes: the package is to reserve space for other specs, the class is to avoid conflict with servers. > The JEP does make a proposal on the module and package name, but I think > a discussion on naming will be needed. I'm not sure that we need to > decide that now, but your point has been noted. OK. -- - DML From james at lightbend.com Fri Dec 8 00:30:32 2017 From: james at lightbend.com (James Roper) Date: Fri, 8 Dec 2017 11:30:32 +1100 Subject: JEP 321: HTTP Client - Use of Flow.Subscriber and Flow.Publisher Message-ID: Hi all, I wanted to start a discussion about the use of Flow.Subscriber and Flow.Publisher in JEP 321 (HTTP Client API). It seems that users are required to implement their own publishers and subscribers, that is, they can't take a Flow.Publisher or Flow.Subscriber provided by another reactive streams implementation, and pass it on to the HttpClient API. The reason for this is that the HttpClient API doesn't accept Flow.Publisher/Flow.Subscriber, rather it extends them in HttpRequest.BodyPublisher and HttpResponse.BodySubscriber, and then requires the user to return instances of those sub interfaces from their BodyHandlers. Let's say I have a database driver that produces a Flow.Subscriber for consuming a stream to be stored in the database, and I want to plumb a response body into that database subscriber. I can't return this from an HttpResponse.BodyHandler it requires me to return a HttpResponse.BodySubscriber, not a Flow.Subscriber. Of course, users can implement their own HttpResponse.BodySubscriber that delegates onSubscribe/onNext/onComplete/onError to the databases Flow.Subscriber, but needing to do this every time does not provide a great developer experience. In order for reactive streams implementations to integrate with each other, Flow.Subscriber and Flow.Publisher should only be extended if the implementation is providing its own implementation of that interface and doesn't expect end users to implement it. For cases where an implementation expects users either to pass that sub interface, or return it from a method to that they implement as is the case for BodyHandler, then it has to be a Flow.Subscriber or Flow.Publisher, not a sub interface. So perhaps HttpResponse.BodySubscriber should be modified to, rather than extending Flow.Subscriber, have a getSubscriber() method that returns a Flow.Subscriber. Alternatively, the API could be inverted, for example, HttpResponse.BodyHandler.apply could be modified to take three parameters, the status code, the HttpHeaders, and a Flow.Publisher, and the return value could be CompletionStage. This approach would actually maximise interoperability, since a lot of reactive streams implementations are publisher centric. For example, to my knowledge RxJava only provides limited support for creating and transforming data using a Flow.Subscriber, whereas if you give RxJava a Flow.Publisher, you can then do the full range of operations (eg map) that RxJava supports on that stream (someone with a better understanding of RxJava correct me if I'm wrong). There's probably many other solutions, these are just two that I've thought of. There is also a broader discussion that needs to be had in the reactive streams community over whether there should be a bias towards everything returning/accepting publishers, or if implementations should evenly support both. Supporting one or the other arbitrarily will have interoperability implications, and while it's definitely beyond the scope of JEP 321 to decide on that, it's something that we should keep in mind. Regards, James -- *James Roper* *Senior Octonaut* Lightbend ? Build reactive apps! Twitter: @jroper -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Fri Dec 8 17:31:55 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 8 Dec 2017 17:31:55 +0000 Subject: JEP 321: HTTP Client - Use of Flow.Subscriber and Flow.Publisher In-Reply-To: References: Message-ID: <3f03925a-419c-4b90-7fd8-7b8da4104485@oracle.com> James, Thanks for taking the time to look at this, and sending your thoughts. On 08/12/17 00:30, James Roper wrote: > Hi all, > > I wanted to start a discussion about the use of Flow.Subscriber and > Flow.Publisher in JEP 321 (HTTP Client API). > > It seems that users are required to implement their own publishers and > subscribers, that is, they can't take a Flow.Publisher or > Flow.Subscriber provided by another reactive streams implementation, and > pass it on to the HttpClient API. The reason for this is that the > HttpClient API doesn't accept Flow.Publisher/Flow.Subscriber, rather it > extends them in HttpRequest.BodyPublisher and > HttpResponse.BodySubscriber, and then requires the user to return > instances of those sub interfaces from their BodyHandlers. ... Great point. I think we can address this with straight forward adapters. For example: public interface BodyPublisher extends Flow.Publisher { /** * Returns a request body publisher whose body is retrieved from the * given {@code Flow.Publisher}. The returned request body publisher * has an unknown content length. * * @apiNote This method can be used as an adapter between {@code * BodyPublisher} and {@code Flow.Publisher}. * * @param publisher the publisher responsible for publishing the body * @return a BodyPublisher */ static BodyPublisher fromPublisher(Flow.Publisher publisher) { ... } ... public BodySubscriber apply(int statusCode, HttpHeaders responseHeaders); /** * Returns a response body handler that returns a {@link BodySubscriber * BodySubscriber}{@code } obtained from {@link * BodySubscriber#fromSubscriber(Subscriber)}. * * @apiNote This method can be used as an adapter between {@code * BodySubscriber} and {@code Flow.Subscriber}. * *

For example: * 

 {@code
       * TextSubscriber subscriber = ...;  // accumulates bytes and 
transforms them into a String.
       * Supplier result = subscriber::getTextResult;
       *
       * CompletableFuture cf =  client
       *         .sendAsync(request, BodyHandler.fromSubscriber(subscriber))
       *         .thenApply((response -> result.get()));
       * String text = cf.join();
       * }
* * @param subscriber the subscriber * @return a response body handler */ public static BodyHandler fromSubscriber(Subscriber> subscriber) { ... } // Add an equivalent BodySubscriber ... This would allow the API to retain its Flow specific types ( that add additional HTTP specific and API behavior ), while interacting, without much fuss, with regular Publishers and Subscribers. -Chris. From james at lightbend.com Mon Dec 11 00:47:22 2017 From: james at lightbend.com (James Roper) Date: Mon, 11 Dec 2017 11:47:22 +1100 Subject: JEP 321: HTTP Client - Use of Flow.Subscriber and Flow.Publisher In-Reply-To: <3f03925a-419c-4b90-7fd8-7b8da4104485@oracle.com> References: <3f03925a-419c-4b90-7fd8-7b8da4104485@oracle.com> Message-ID: Hi Chris, This looks like a straight forward way to solve the problem with minimal disruption from the existing API. Can I make a few suggestions though? We could add a contentLength parameter to fromPublisher, to allow Flow.Publishers where the content length is known to be easily converted to BodyPublisher: static BodyPublisher fromPublisher(Flow.Publisher publisher, int contentLength) { ... } This would mean if you were receiving a servlet request body and publishing it to another location, then you could do something like this (this uses a reactive streams implementation on top of the servlet API that I wrote): HttpServletRequest request = ... long contentLength = -1; if (request.getHeader("Content-Length") != null) { contentLength = Long.parseLong(request.getHeader("Content-Length")); } Publisher publisher = new RequestPublisher(request.startAsync(), 8192); HttpRequest clientRequest = HttpRequest.newBuilder(target) .POST(BodyPublisher.fromPublisher(publisher, contentLength)) .build() Perhaps the method could be overloaded for both supplying and not supplying a content length. Similarly, I think a fromSubscriber API that accepted a CompletionStage would be a little more fluent than having to supply it externally: public static BodyHandler fromSubscriber(Subscriber> subscriber, CompletionStage bodyFuture) { ... } Then you could have something like this: TextSubscriber subscriber = ...; // accumulates bytes and transforms them into a CompletionStage. CompletionStage result = subscriber.getTextResult(); CompletableFuture cf = client .sendAsync(request, BodyHandler.fromSubscriber(subscriber, result)); String text = cf.join(); Likewise, this could be an overload of fromSubscriber if we want the option of not specifying a body future. One thing I think needs to be carefully specified is, if the method doesn't accept a CompletionStage, when/how the CompletionStage returned from send is redeemed. Regards, James On 9 December 2017 at 04:31, Chris Hegarty wrote: > James, > > Thanks for taking the time to look at this, and sending your thoughts. > > On 08/12/17 00:30, James Roper wrote: > > Hi all, > > > > I wanted to start a discussion about the use of Flow.Subscriber and > > Flow.Publisher in JEP 321 (HTTP Client API). > > > > It seems that users are required to implement their own publishers and > > subscribers, that is, they can't take a Flow.Publisher or > > Flow.Subscriber provided by another reactive streams implementation, and > > pass it on to the HttpClient API. The reason for this is that the > > HttpClient API doesn't accept Flow.Publisher/Flow.Subscriber, rather it > > extends them in HttpRequest.BodyPublisher and > > HttpResponse.BodySubscriber, and then requires the user to return > > instances of those sub interfaces from their BodyHandlers. ... > > Great point. I think we can address this with straight forward adapters. > For example: > > public interface BodyPublisher extends Flow.Publisher { > > /** > * Returns a request body publisher whose body is retrieved from the > * given {@code Flow.Publisher}. The returned request body publisher > * has an unknown content length. > * > * @apiNote This method can be used as an adapter between {@code > * BodyPublisher} and {@code Flow.Publisher}. > * > * @param publisher the publisher responsible for publishing the body > * @return a BodyPublisher > */ > static BodyPublisher fromPublisher(Flow.Publisher > publisher) { > ... > } > > ... > > public BodySubscriber apply(int statusCode, HttpHeaders > responseHeaders); > > /** > * Returns a response body handler that returns a {@link > BodySubscriber > * BodySubscriber}{@code } obtained from {@link > * BodySubscriber#fromSubscriber(Subscriber)}. > * > * @apiNote This method can be used as an adapter between {@code > * BodySubscriber} and {@code Flow.Subscriber}. > * > *

For example: > * 

 {@code
>       * TextSubscriber subscriber = ...;  // accumulates bytes and
> transforms them into a String.
>       * Supplier result = subscriber::getTextResult;
>       *
>       * CompletableFuture cf =  client
>       *         .sendAsync(request, BodyHandler.fromSubscriber(sub
> scriber))
>       *         .thenApply((response -> result.get()));
>       * String text = cf.join();
>       * }
> * > * @param subscriber the subscriber > * @return a response body handler > */ > public static BodyHandler fromSubscriber(Subscriber List> subscriber) { > ... > } > > // Add an equivalent BodySubscriber ... > > > This would allow the API to retain its Flow specific types ( that add > additional HTTP specific and API behavior ), while interacting, without > much fuss, with regular Publishers and Subscribers. > > -Chris. > -- *James Roper* *Senior Octonaut* Lightbend ? Build reactive apps! Twitter: @jroper -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Mon Dec 11 15:48:30 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 11 Dec 2017 15:48:30 +0000 Subject: JEP 321: HTTP Client - Use of Flow.Subscriber and Flow.Publisher In-Reply-To: References: <3f03925a-419c-4b90-7fd8-7b8da4104485@oracle.com> Message-ID: James, On 11/12/17 00:47, James Roper wrote: > Hi Chris, > > This looks like a straight forward way to solve the problem with minimal > disruption from the existing API. Can I make a few suggestions though? Of course, your input here is much appreciated. > We could add a contentLength parameter to fromPublisher, to allow > Flow.Publishers where the content length is known to be easily converted > to BodyPublisher: > > static BodyPublisher fromPublisher(Flow.Publisher publisher, > int contentLength) { > ... > } Good idea. Added ( as can be seen below ). > This would mean if you were receiving a servlet request body and > publishing it to another location, then you could do something like this > (this uses a reactive streams implementation on top of the servlet API > that I wrote): > > HttpServletRequest request = ... > long contentLength = -1; > if (request.getHeader("Content-Length") != null) { > contentLength = Long.parseLong(request.getHeader("Content-Length")); > } > Publisher publisher = new > RequestPublisher(request.startAsync(), 8192); > > HttpRequest clientRequest = HttpRequest.newBuilder(target) > .POST(BodyPublisher.fromPublisher(publisher, contentLength)) > .build() Nice. > Perhaps the method could be overloaded for both supplying and not > supplying a content length. I think an overload is justified here. Added. > Similarly, I think a fromSubscriber API that accepted a > CompletionStage would be a little more fluent than having to supply > it externally: Daniel and I discussed this too, but I opted to leave it out initially for simplicity. I think if we have two overloads, then the simple case can still be supported with little ceremony, while allowing a more powerful variant. > public static BodyHandler fromSubscriber(Subscriber List> subscriber, CompletionStage bodyFuture) { > ... > } > > Then you could have something like this: > > TextSubscriber subscriber = ...; // accumulates bytes and transforms > them into a CompletionStage. > CompletionStage result = subscriber.getTextResult(); > > CompletableFuture cf = client > .sendAsync(request, BodyHandler.fromSubscriber(subscriber, result)); > String text = cf.join(); > > Likewise, this could be an overload of fromSubscriber if we want the > option of not specifying a body future. > > One thing I think needs to be carefully specified is, if the method > doesn't accept a CompletionStage, when/how the CompletionStage returned > from send is redeemed. Hmmm... this could be tricky. I think we can avoid the scenario completely by accepting a finishing function that can generate/return the value to use for completion, rather than the CF itself. Here is an outline of all of the above: public interface BodyPublisher extends Flow.Publisher { /** * Returns a request body publisher whose body is retrieved from the * given {@code Flow.Publisher}. The returned request body publisher * has an unknown content length. * * @apiNote This method can be used as an adapter between {@code * BodyPublisher} and {@code Flow.Publisher}, where the amount of * request body that the publisher will publish is unknown. * * @param publisher the publisher responsible for publishing the body * @return a BodyPublisher */ static BodyPublisher fromPublisher(Flow.Publisher publisher) { ... } /** * Returns a request body publisher whose body is retrieved from the * given {@code Flow.Publisher}. The returned request body publisher * has the given content length. * *

The given {@code contentLength} is a positive number, that * represents the exact amount of bytes the {@code publisher} must * publish. * * @apiNote This method can be used as an adapter between {@code * BodyPublisher} and {@code Flow.Publisher}, where the amount of * request body that the publisher will publish is known. * * @param publisher the publisher responsible for publishing the body * @param contentLength a positive number representing the exact * amount of bytes the publisher will publish * @throws IllegalArgumentException if the content length is * non-positive * @return a BodyPublisher */ static BodyPublisher fromPublisher(Flow.Publisher publisher, long contentLength) { ... } public interface BodyHandler { /** * Returns a response body handler that returns a {@link BodySubscriber * BodySubscriber}{@code } obtained from {@linkplain * BodySubscriber#fromSubscriber(Subscriber)}, with the given * {@code subscriber}. * *

The response body is not available through this, or the {@code * HttpResponse} API, but instead all response body is forwarded to the * given {@code subscriber}, which should make it available, if * appropriate, through some other mechanism, e.g. an entry in a * database, etc. * * @apiNote This method can be used as an adapter between {@code * BodySubscriber} and {@code Flow.Subscriber}. * *

For example: * 

 {@code
      *  TextSubscriber subscriber = new TextSubscriber();
      *  HttpResponse response = client.sendAsync(request,
      *      BodyHandler.fromSubscriber(subscriber)).join();
      *  System.out.println(response.statusCode());
      * }
* * @param subscriber the subscriber * @return a response body handler */ public static BodyHandler fromSubscriber(Subscriber> subscriber) { ... } /** * Returns a response body handler that returns a {@link BodySubscriber * BodySubscriber}{@code } obtained from {@link * BodySubscriber#fromSubscriber(Subscriber, Function)}, with the * given {@code subscriber} and {@code finisher} function. * *

The given {@code finisher} function is applied after the given * subscriber's {@code onComplete} has been invoked. The {@code finisher} * function is invoked with the given subscriber, and returns a value * that is set as the response's body. * * @apiNote This method can be used as an adapter between {@code * BodySubscriber} and {@code Flow.Subscriber}. * *

For example: * 

 {@code
      * TextSubscriber subscriber = ...;  // accumulates bytes and 
transforms them into a String.*
      * HttpResponse response = client.sendAsync(request,
      *     BodyHandler.fromSubscriber(subscriber, 
TextSubscriber::getTextResult)).join();
      * String text = response.body();
      * }
* * @param the type of the Subscriber * @param the type of the response body * @param subscriber the subscriber * @param finisher a function to be applied after the subscriber has completed * @return a response body handler */ public static >,T> BodyHandler fromSubscriber(S subscriber, Function finisher) { ... } // And a similar pair of BodySubscriber methods, omitted for brevity. -Chris. From martinrb at google.com Mon Dec 11 17:24:03 2017 From: martinrb at google.com (Martin Buchholz) Date: Mon, 11 Dec 2017 09:24:03 -0800 Subject: RFR: 8193034: Optimize URL.toExternalForm In-Reply-To: <8B8F59AA-A5B3-4173-9B1F-7A726D8C609D@oracle.com> References: <8B8F59AA-A5B3-4173-9B1F-7A726D8C609D@oracle.com> Message-ID: There's now a Martin-style benchmark at http://cr.openjdk.java.net/~martin/webrevs/jdk/URLMicroBenchmark/ that suggests the code is ~ 25% faster with default JVM flags (C2) but ~ 25% slower with C1, as you might expect with multiple String concatenation. I think we want to optimize for the default and assume that multiple String concatenation does in fact get optimized. I'm not planning to check in the benchmark code. On Tue, Dec 5, 2017 at 2:10 AM, Chris Hegarty wrote: > > > On 5 Dec 2017, at 04:01, Martin Buchholz wrote: > > > > http://cr.openjdk.java.net/~martin/webrevs/jdk/URL.toExternalForm/ > > Since the motivation for this change is performance, do you have any > performance numbers / benchmarks that you can share? > > -Chris. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.x.mcmahon at oracle.com Tue Dec 12 10:07:26 2017 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Tue, 12 Dec 2017 10:07:26 +0000 Subject: RFR: 8192966 HttpClient should reuse TCP connection for h2c connections Message-ID: <5A2FAA5E.9000909@oracle.com> http://cr.openjdk.java.net/~michaelm/8192966/webrev.1/ Thanks, Michael. From chris.hegarty at oracle.com Tue Dec 12 10:27:04 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 12 Dec 2017 10:27:04 +0000 Subject: RFR 8185027: Typo in java.net.URLClassLoader.findResources(String) method documentation Message-ID: <04935425-a2e7-150e-589b-82f59991ad1d@oracle.com> The returns section of the java.net.URLClassLoader.findResources(String) method documentation, contains the following sentence: "an Enumeration of URLs If the loader is closed, the Enumeration will be empty." should be replaced by: "An Enumeration of URLs. If the loader is closed, the Enumeration will be empty." diff --git a/src/java.base/share/classes/java/net/URLClassLoader.java b/src/java.base/share/classes/java/net/URLClassLoader.java --- a/src/java.base/share/classes/java/net/URLClassLoader.java +++ b/src/java.base/share/classes/java/net/URLClassLoader.java @@ -658,7 +658,7 @@ * * @param name the resource name * @exception IOException if an I/O exception occurs - * @return an {@code Enumeration} of {@code URL}s + * @return An {@code Enumeration} of {@code URL}s. * If the loader is closed, the Enumeration will be empty. */ public Enumeration findResources(final String name) -Chris. From chris.hegarty at oracle.com Tue Dec 12 10:30:27 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 12 Dec 2017 10:30:27 +0000 Subject: RFR: 8193034: Optimize URL.toExternalForm In-Reply-To: References: <8B8F59AA-A5B3-4173-9B1F-7A726D8C609D@oracle.com> Message-ID: Martin, I think your changes are good. Thanks for sharing your benchmark and results. -Chris. On 11/12/17 17:24, Martin Buchholz wrote: > There's now a Martin-style benchmark at > http://cr.openjdk.java.net/~martin/webrevs/jdk/URLMicroBenchmark/ > that suggests the code is ~ 25% faster with default JVM flags (C2) but ~ > 25% slower with C1, as you might expect with multiple String > concatenation. I think we want to optimize for the default and assume > that multiple String concatenation does in fact get optimized. > I'm not planning to check in the benchmark code. > > On Tue, Dec 5, 2017 at 2:10 AM, Chris Hegarty > wrote: > > > > On 5 Dec 2017, at 04:01, Martin Buchholz > wrote: > > > > > http://cr.openjdk.java.net/~martin/webrevs/jdk/URL.toExternalForm/ > > > Since the motivation for this change is performance, do you have any > performance numbers / benchmarks that you can share? > > -Chris. > > From daniel.fuchs at oracle.com Tue Dec 12 10:32:26 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 12 Dec 2017 10:32:26 +0000 Subject: RFR: 8192966 HttpClient should reuse TCP connection for h2c connections In-Reply-To: <5A2FAA5E.9000909@oracle.com> References: <5A2FAA5E.9000909@oracle.com> Message-ID: <6b207569-5587-c26c-c4db-d7ecb8600de1@oracle.com> Hi Michael, I wonder whether Http2Connection::closeStream would be a better place to call Stream::checkConnectionClosure? If not then shouldn't it be called in Stream::release as well? best regards, -- daniel On 12/12/2017 10:07, Michael McMahon wrote: > > http://cr.openjdk.java.net/~michaelm/8192966/webrev.1/ > > Thanks, > Michael. From Alan.Bateman at oracle.com Tue Dec 12 10:36:45 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 12 Dec 2017 10:36:45 +0000 Subject: RFR 8185027: Typo in java.net.URLClassLoader.findResources(String) method documentation In-Reply-To: <04935425-a2e7-150e-589b-82f59991ad1d@oracle.com> References: <04935425-a2e7-150e-589b-82f59991ad1d@oracle.com> Message-ID: On 12/12/2017 10:27, Chris Hegarty wrote: > The returns section of the java.net.URLClassLoader.findResources(String) > method documentation, contains the following sentence: > ?? "an Enumeration of URLs If the loader is closed, the Enumeration > ??? will be empty." > should be replaced by: > ?? "An Enumeration of URLs. If the loader is closed, the Enumeration > ??? will be empty." Looks fine. Separately, I don't know if "empty" is defined anywhere for Enumerations, it could just say "contains no elements" or other variants. -Alan From chris.hegarty at oracle.com Tue Dec 12 10:40:03 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 12 Dec 2017 10:40:03 +0000 Subject: RFR 8185027: Typo in java.net.URLClassLoader.findResources(String) method documentation In-Reply-To: References: <04935425-a2e7-150e-589b-82f59991ad1d@oracle.com> Message-ID: On 12/12/17 10:36, Alan Bateman wrote: >... > Looks fine. Separately, I don't know if "empty" is defined anywhere for > Enumerations, it could just say "contains no elements" or other variants. That is better. diff --git a/src/java.base/share/classes/java/net/URLClassLoader.java b/src/java.base/share/classes/java/net/URLClassLoader.java --- a/src/java.base/share/classes/java/net/URLClassLoader.java +++ b/src/java.base/share/classes/java/net/URLClassLoader.java @@ -658,8 +658,8 @@ * * @param name the resource name * @exception IOException if an I/O exception occurs - * @return an {@code Enumeration} of {@code URL}s - * If the loader is closed, the Enumeration will be empty. + * @return An {@code Enumeration} of {@code URL}s. + * If the loader is closed, the Enumeration contains no elements. */ public Enumeration findResources(final String name) throws IOException -Chris. From Alan.Bateman at oracle.com Tue Dec 12 10:43:31 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 12 Dec 2017 10:43:31 +0000 Subject: RFR 8185027: Typo in java.net.URLClassLoader.findResources(String) method documentation In-Reply-To: References: <04935425-a2e7-150e-589b-82f59991ad1d@oracle.com> Message-ID: On 12/12/2017 10:40, Chris Hegarty wrote: > > -???? * @return an {@code Enumeration} of {@code URL}s > -???? *???????? If the loader is closed, the Enumeration will be empty. > +???? * @return An {@code Enumeration} of {@code URL}s. > +???? *???????? If the loader is closed, the Enumeration contains no > elements. Looks good. From chris.hegarty at oracle.com Tue Dec 12 11:13:02 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 12 Dec 2017 11:13:02 +0000 Subject: JEP 321: HTTP Client - Use of Flow.Subscriber and Flow.Publisher In-Reply-To: References: <3f03925a-419c-4b90-7fd8-7b8da4104485@oracle.com> Message-ID: I filed the following JIRA issue to track this discussion and proposal. https://bugs.openjdk.java.net/browse/JDK-8193365 I'll start the process of bringing it into 10, unless there are any final comments. FTR, I'm happy where we ended up on this. -Chris. On 11/12/17 15:48, Chris Hegarty wrote: > James, > > On 11/12/17 00:47, James Roper wrote: > > Hi Chris, > > > > This looks like a straight forward way to solve the problem with minimal > > disruption from the existing API. Can I make a few suggestions though? > > Of course, your input here is much appreciated. > > > We could add a contentLength parameter to fromPublisher, to allow > > Flow.Publishers where the content length is known to be easily converted > > to BodyPublisher: > > > > static BodyPublisher fromPublisher(Flow.Publisher publisher, > > int contentLength) { > > ... > > } > > Good idea. Added ( as can be seen below ). > > > This would mean if you were receiving a servlet request body and > > publishing it to another location, then you could do something like this > > (this uses a reactive streams implementation on top of the servlet API > > that I wrote): > > > > HttpServletRequest request = ... > > long contentLength = -1; > > if (request.getHeader("Content-Length") != null) { > > contentLength = Long.parseLong(request.getHeader("Content-Length")); > > } > > Publisher publisher = new > > RequestPublisher(request.startAsync(), 8192); > > > > HttpRequest clientRequest = HttpRequest.newBuilder(target) > > .POST(BodyPublisher.fromPublisher(publisher, contentLength)) > > .build() > > Nice. > > > Perhaps the method could be overloaded for both supplying and not > > supplying a content length. > > I think an overload is justified here. Added. > > > Similarly, I think a fromSubscriber API that accepted a > > CompletionStage would be a little more fluent than having to supply > > it externally: > > Daniel and I discussed this too, but I opted to leave it out initially > for simplicity. I think if we have two overloads, then the simple case > can still be supported with little ceremony, while allowing a more > powerful variant. > > > public static BodyHandler fromSubscriber(Subscriber > List> subscriber, CompletionStage bodyFuture) { > > ... > > } > > > > Then you could have something like this: > > > > TextSubscriber subscriber = ...; // accumulates bytes and transforms > > them into a CompletionStage. > > CompletionStage result = subscriber.getTextResult(); > > > > CompletableFuture cf = client > > .sendAsync(request, BodyHandler.fromSubscriber(subscriber, result)); > > String text = cf.join(); > > > > Likewise, this could be an overload of fromSubscriber if we want the > > option of not specifying a body future. > > > > One thing I think needs to be carefully specified is, if the method > > doesn't accept a CompletionStage, when/how the CompletionStage returned > > from send is redeemed. > > Hmmm... this could be tricky. I think we can avoid the scenario > completely by accepting a finishing function that can generate/return > the value to use for completion, rather than the CF itself. > > Here is an outline of all of the above: > > public interface BodyPublisher extends Flow.Publisher { > > /** > * Returns a request body publisher whose body is retrieved from the > * given {@code Flow.Publisher}. The returned request body publisher > * has an unknown content length. > * > * @apiNote This method can be used as an adapter between {@code > * BodyPublisher} and {@code Flow.Publisher}, where the amount of > * request body that the publisher will publish is unknown. > * > * @param publisher the publisher responsible for publishing the body > * @return a BodyPublisher > */ > static BodyPublisher fromPublisher(Flow.Publisher ByteBuffer> publisher) { ... } > > /** > * Returns a request body publisher whose body is retrieved from the > * given {@code Flow.Publisher}. The returned request body publisher > * has the given content length. > * > *

The given {@code contentLength} is a positive number, that > * represents the exact amount of bytes the {@code publisher} must > * publish. > * > * @apiNote This method can be used as an adapter between {@code > * BodyPublisher} and {@code Flow.Publisher}, where the amount of > * request body that the publisher will publish is known. > * > * @param publisher the publisher responsible for publishing the body > * @param contentLength a positive number representing the exact > * amount of bytes the publisher will publish > * @throws IllegalArgumentException if the content length is > * non-positive > * @return a BodyPublisher > */ > static BodyPublisher fromPublisher(Flow.Publisher ByteBuffer> publisher, > long contentLength) { ... } > > > public interface BodyHandler { > > /** > * Returns a response body handler that returns a {@link > BodySubscriber > * BodySubscriber}{@code } obtained from {@linkplain > * BodySubscriber#fromSubscriber(Subscriber)}, with the given > * {@code subscriber}. > * > *

The response body is not available through this, or the {@code > * HttpResponse} API, but instead all response body is forwarded to > the > * given {@code subscriber}, which should make it available, if > * appropriate, through some other mechanism, e.g. an entry in a > * database, etc. > * > * @apiNote This method can be used as an adapter between {@code > * BodySubscriber} and {@code Flow.Subscriber}. > * > *

For example: > * 

 {@code
>       *  TextSubscriber subscriber = new TextSubscriber();
>       *  HttpResponse response = client.sendAsync(request,
>       *      BodyHandler.fromSubscriber(subscriber)).join();
>       *  System.out.println(response.statusCode());
>       * }
> * > * @param subscriber the subscriber > * @return a response body handler > */ > public static BodyHandler > fromSubscriber(Subscriber> subscriber) { > ... } > > /** > * Returns a response body handler that returns a {@link > BodySubscriber > * BodySubscriber}{@code } obtained from {@link > * BodySubscriber#fromSubscriber(Subscriber, Function)}, with the > * given {@code subscriber} and {@code finisher} function. > * > *

The given {@code finisher} function is applied after the given > * subscriber's {@code onComplete} has been invoked. The {@code > finisher} > * function is invoked with the given subscriber, and returns a value > * that is set as the response's body. > * > * @apiNote This method can be used as an adapter between {@code > * BodySubscriber} and {@code Flow.Subscriber}. > * > *

For example: > * 

 {@code
>       * TextSubscriber subscriber = ...;  // accumulates bytes and 
> transforms them into a String.*
>       * HttpResponse response = client.sendAsync(request,
>       *     BodyHandler.fromSubscriber(subscriber, 
> TextSubscriber::getTextResult)).join();
>       * String text = response.body();
>       * }
> * > * @param the type of the Subscriber > * @param the type of the response body > * @param subscriber the subscriber > * @param finisher a function to be applied after the subscriber > has completed > * @return a response body handler > */ > public static >,T> > BodyHandler > fromSubscriber(S subscriber, Function finisher) { ... } > > > // And a similar pair of BodySubscriber methods, omitted for brevity. > > > -Chris. From Alan.Bateman at oracle.com Tue Dec 12 11:56:42 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 12 Dec 2017 11:56:42 +0000 Subject: RFR: 8193034: Optimize URL.toExternalForm In-Reply-To: References: <8B8F59AA-A5B3-4173-9B1F-7A726D8C609D@oracle.com> Message-ID: <70cdfa0a-744d-560f-05ab-e623eb54a9d5@oracle.com> On 11/12/2017 17:24, Martin Buchholz wrote: > There's now a Martin-style benchmark at > http://cr.openjdk.java.net/~martin/webrevs/jdk/URLMicroBenchmark/ > > that suggests the code is ~ 25% faster with default JVM flags (C2) but > ~ 25% slower with C1, as you might expect with multiple String > concatenation.? I think we want to optimize for the default and assume > that multiple String concatenation does in fact get optimized. > I'm not planning to check in the benchmark code. > Good to confirm the improvement, I think the change is good to push. -Alan From michael.x.mcmahon at oracle.com Tue Dec 12 13:32:53 2017 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Tue, 12 Dec 2017 13:32:53 +0000 Subject: RFR: 8192966 HttpClient should reuse TCP connection for h2c connections In-Reply-To: <6b207569-5587-c26c-c4db-d7ecb8600de1@oracle.com> References: <5A2FAA5E.9000909@oracle.com> <6b207569-5587-c26c-c4db-d7ecb8600de1@oracle.com> Message-ID: <5A2FDA85.5070203@oracle.com> Thanks Daniel. I'm looking at this change again as I have noticed the behavior when multiple https requests are initiated might not be what we want. I'll post another webrev later. - Michael On 12/12/2017, 10:32, Daniel Fuchs wrote: > Hi Michael, > > I wonder whether Http2Connection::closeStream would be > a better place to call Stream::checkConnectionClosure? > If not then shouldn't it be called in Stream::release > as well? > > best regards, > > -- daniel > > On 12/12/2017 10:07, Michael McMahon wrote: >> >> http://cr.openjdk.java.net/~michaelm/8192966/webrev.1/ >> >> Thanks, >> Michael. > From daniel.fuchs at oracle.com Tue Dec 12 17:43:24 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 12 Dec 2017 17:43:24 +0000 Subject: RFR: 8193370 Provide more user friendly defaults for HTTP/2 client settings Message-ID: <6b369614-5dbd-899b-76b4-0b4affaf5822@oracle.com> Hi, please find below a patch for: 8193370 Provide more user friendly defaults for HTTP/2 client settings https://bugs.openjdk.java.net/browse/JDK-8193370 webrev: http://cr.openjdk.java.net/~dfuchs/webrev_8193370/webrev.01/ best regards, -- daniel From chris.hegarty at oracle.com Tue Dec 12 18:04:31 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Tue, 12 Dec 2017 18:04:31 +0000 Subject: RFR: 8193370 Provide more user friendly defaults for HTTP/2 client settings In-Reply-To: <6b369614-5dbd-899b-76b4-0b4affaf5822@oracle.com> References: <6b369614-5dbd-899b-76b4-0b4affaf5822@oracle.com> Message-ID: <32CAC069-921E-41A2-A726-C8196D467143@oracle.com> > On 12 Dec 2017, at 17:43, Daniel Fuchs wrote: > > Hi, > > please find below a patch for: > > 8193370 Provide more user friendly defaults for HTTP/2 client settings > https://bugs.openjdk.java.net/browse/JDK-8193370 > > webrev: > http://cr.openjdk.java.net/~dfuchs/webrev_8193370/webrev.01/ This looks good Daniel. It is good that you have added restrictions on the allowable values for some of these settings, some of which are specified by the RFC. I also agree with the increase is window sizes. This should improve overall throughput and reduce network trashing from windows updates. -Chris. From simone.bordet at gmail.com Tue Dec 12 18:21:55 2017 From: simone.bordet at gmail.com (Simone Bordet) Date: Tue, 12 Dec 2017 19:21:55 +0100 Subject: RFR: 8193370 Provide more user friendly defaults for HTTP/2 client settings In-Reply-To: <32CAC069-921E-41A2-A726-C8196D467143@oracle.com> References: <6b369614-5dbd-899b-76b4-0b4affaf5822@oracle.com> <32CAC069-921E-41A2-A726-C8196D467143@oracle.com> Message-ID: Hi, On Tue, Dec 12, 2017 at 7:04 PM, Chris Hegarty wrote: > >> On 12 Dec 2017, at 17:43, Daniel Fuchs wrote: >> >> Hi, >> >> please find below a patch for: >> >> 8193370 Provide more user friendly defaults for HTTP/2 client settings >> https://bugs.openjdk.java.net/browse/JDK-8193370 >> >> webrev: >> http://cr.openjdk.java.net/~dfuchs/webrev_8193370/webrev.01/ > > This looks good Daniel. It is good that you have added restrictions > on the allowable values for some of these settings, some of which > are specified by the RFC. I also agree with the increase is window > sizes. This should improve overall throughput and reduce network > trashing from windows updates. Do I read it correctly that the default session window is 1 GiB ? This would impose on the client a large memory requirement especially when connecting to multiple domains. I would suggest smaller values. Our experience is that even in fast network (not localhost) 16 MiB session windows will hit TCP congestion control before stalling HTTP/2 flow control. -- Simone Bordet --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From daniel.fuchs at oracle.com Tue Dec 12 21:02:20 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 12 Dec 2017 21:02:20 +0000 Subject: RFR: 8193370 Provide more user friendly defaults for HTTP/2 client settings In-Reply-To: References: <6b369614-5dbd-899b-76b4-0b4affaf5822@oracle.com> <32CAC069-921E-41A2-A726-C8196D467143@oracle.com> Message-ID: <618aff5c-8d5a-7f16-a800-1e20909ff1cb@oracle.com> Hi Simone, Long time no talk ;-) Thanks for sharing your experience! On 12/12/2017 18:21, Simone Bordet wrote: > Our experience is that even in fast network (not localhost) 16 MiB > session windows will hit TCP congestion control before stalling HTTP/2 > flow control. I have experimented with 16Mib and 32Mib. Even on slow network (VPN) the settings in my patch give me a boost of 2+ times faster response. These settings can be overridden by specifying an alternate value for the corresponding net/system properties, if the defaults is not appropriate for the application. If I'm not mistaken curl seems also to be using large connection window sizes. best regards, -- daniel From daniel.fuchs at oracle.com Wed Dec 13 01:17:58 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Wed, 13 Dec 2017 01:17:58 +0000 Subject: RFR: 8193370 Provide more user friendly defaults for HTTP/2 client settings In-Reply-To: <618aff5c-8d5a-7f16-a800-1e20909ff1cb@oracle.com> References: <6b369614-5dbd-899b-76b4-0b4affaf5822@oracle.com> <32CAC069-921E-41A2-A726-C8196D467143@oracle.com> <618aff5c-8d5a-7f16-a800-1e20909ff1cb@oracle.com> Message-ID: <65e52bf1-6629-7d72-456f-cc812b5b9ffe@oracle.com> Hi Simone, On 12/12/2017 21:02, Daniel Fuchs wrote: > On 12/12/2017 18:21, Simone Bordet wrote: >> Our experience is that even in fast network (not localhost) 16 MiB >> session windows will hit TCP congestion control before stalling HTTP/2 >> flow control. > > I have experimented with 16Mib and 32Mib. > Even on slow network (VPN) the settings in my patch give me a > boost of 2+ times faster response. Chris & I played some more with different settings. I may have been too aggressive in my previous patch :-) It seems indeed that a session windows of 32M is a reasonable default. I have updated my webrev and done some intensive same-binary test runs: http://cr.openjdk.java.net/~dfuchs/webrev_8193370/webrev.02 best regards, -- daniel From chris.hegarty at oracle.com Wed Dec 13 11:36:52 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 13 Dec 2017 11:36:52 +0000 Subject: RFR: 8193370 Provide more user friendly defaults for HTTP/2 client settings In-Reply-To: <65e52bf1-6629-7d72-456f-cc812b5b9ffe@oracle.com> References: <6b369614-5dbd-899b-76b4-0b4affaf5822@oracle.com> <32CAC069-921E-41A2-A726-C8196D467143@oracle.com> <618aff5c-8d5a-7f16-a800-1e20909ff1cb@oracle.com> <65e52bf1-6629-7d72-456f-cc812b5b9ffe@oracle.com> Message-ID: <9a00bb1e-5f25-9869-6c1d-06030c19c0f6@oracle.com> Daniel, Simone, On 13/12/17 01:17, Daniel Fuchs wrote: > Hi Simone, > > On 12/12/2017 21:02, Daniel Fuchs wrote: >> On 12/12/2017 18:21, Simone Bordet wrote: >>> Our experience is that even in fast network (not localhost) 16 MiB >>> session windows will hit TCP congestion control before stalling HTTP/2 >>> flow control. >> >> I have experimented with 16Mib and 32Mib. >> Even on slow network (VPN) the settings in my patch give me a >> boost of 2+ times faster response. > > Chris & I played some more with different settings. > I may have been too aggressive in my previous patch :-) > It seems indeed that a session windows of 32M is a reasonable default. Yes, I think this is better, thanks. > I have updated my webrev and done some intensive same-binary > test runs: > > http://cr.openjdk.java.net/~dfuchs/webrev_8193370/webrev.02 This link appears to be not working, so I copied the webrev to the following location: http://cr.openjdk.java.net/~chegar/webrev_8193370/webrev.02/ -Chris. From chris.hegarty at oracle.com Wed Dec 13 15:03:33 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 13 Dec 2017 15:03:33 +0000 Subject: JEP 321: HTTP Client - Use of Flow.Subscriber and Flow.Publisher In-Reply-To: References: <3f03925a-419c-4b90-7fd8-7b8da4104485@oracle.com> Message-ID: <0ea1e0a5-2e43-f58b-426f-2f6469d5e662@oracle.com> On 12/12/17 11:13, Chris Hegarty wrote: > > I filed the following JIRA issue to track this discussion > and proposal. > > https://bugs.openjdk.java.net/browse/JDK-8193365 > > I'll start the process of bringing it into 10, unless there > are any final comments. FTR, I'm happy where we ended up on > this. And link to the CSR that contains the full proposed specification changes: https://bugs.openjdk.java.net/browse/JDK-8193366 -Chris. From andrew_nuss at yahoo.com Thu Dec 14 17:54:34 2017 From: andrew_nuss at yahoo.com (Andy Nuss) Date: Thu, 14 Dec 2017 17:54:34 +0000 (UTC) Subject: does the underlying implementation (not public) of DatagramSocketImp's send receive method busy wait References: <585887490.4621806.1513274074018.ref@mail.yahoo.com> Message-ID: <585887490.4621806.1513274074018@mail.yahoo.com> NOTE: I hope this is not a double-post!? I tried posting before I had joined, not realizing that I had not joined yet. I really tried hard googling.? No success.? I am trying to do an appropriate implementation of a UDP "sender" that sends lots of UDP packets to many different locations.? What I think I should do is open several any-port UDP sockets on this machine, and bind a thread to each one.? The threads all receive byte[] (etc) of non-zero length size, and destination ipaddr tuples in a queue.? Of course, while the queue is empty the threads wait for another addition to the queue.? As the queues of the threads get flooded, they drain and send the udp packets as fast as the can to the appropriate destination ipaddrs. The big question is this: does the underlying implementation of DatagramSocketImpl.send() (whose code I couldn't locate googling, and might be native anyways, I don't know), do busy looping on a flag of the UDP protocol?? I.e. does it optimally leverage java threading, during the byte by byte xfer process and handshaking, so that other java threads can do useful work, or while it is sending a specific packet, does it block all other threads on that cpu? Similarly, in DatagramChannel, it is not clear of blocking mode means that it busy waits internally.? For example, the non-blocking examples of sending DatagramChannel packets say, while the ByteBuffer is not completely sent, do useful work here and then try again.? That's not an easy coding model, because the only reasonable solution is Thread.sleep(for_how_long) between successive attempts to send pieces of the ByteBuffer, and that may not scale to all the UDP traffic being sent. Thus I would hope that I can use DatagramSocket.send(), in a thread of highish priority, and be confident that during the internals of the send impl, its not eating all the cpu time on the core on which that thread is running. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Fri Dec 15 15:37:14 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 15 Dec 2017 15:37:14 +0000 Subject: RFR 8193365: Improve interoperability between HTTP Client's BodyPublisher/BodySubscriber and Flow.Subscriber/Publisher Message-ID: <19cd5458-0851-6988-4d6e-d39f0b295396@oracle.com> This review is for 8193365 [1], to improve interoperability between HTTP Client's BodyPublisher/BodySubscriber and Flow.Subscriber/Publisher, that was raised as part of feedback on JEP 321 [2]. http://cr.openjdk.java.net/~chegar/8193365/webrev.01/ -Chris. [1] https://bugs.openjdk.java.net/browse/JDK-8193365 [2] http://mail.openjdk.java.net/pipermail/net-dev/2017-December/thread.html#11063 From chris.hegarty at oracle.com Mon Dec 18 13:18:51 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Mon, 18 Dec 2017 13:18:51 +0000 Subject: RFR 8193698: Null handling in BodyPublisher, BodyHandler, and BodySubscriber convenience static factory methods Message-ID: The HTTP Client API specifies that a NullPointerException will be thrown when null is passed as an argument, expect where explicitly specified. This issue adds correct null arg checking where necessary, which is needed for spec conformance. http://cr.openjdk.java.net/~chegar/8193698/ -Chris. [1] https://bugs.openjdk.java.net/browse/JDK-8193698 From daniel.fuchs at oracle.com Mon Dec 18 15:32:45 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Mon, 18 Dec 2017 15:32:45 +0000 Subject: RFR 8193698: Null handling in BodyPublisher, BodyHandler, and BodySubscriber convenience static factory methods In-Reply-To: References: Message-ID: <52d8024e-1345-85a6-fffb-407d60abed6e@oracle.com> Looks good to me Chris! -- daniel On 18/12/2017 13:18, Chris Hegarty wrote: > The HTTP Client API specifies that a NullPointerException will > be thrown when null is passed as an argument, expect where > explicitly specified. This issue adds correct null arg checking > where necessary, which is needed for spec conformance. > > http://cr.openjdk.java.net/~chegar/8193698/ > > -Chris. > > [1] https://bugs.openjdk.java.net/browse/JDK-8193698 From pmarks at google.com Mon Dec 18 21:24:28 2017 From: pmarks at google.com (Paul Marks) Date: Mon, 18 Dec 2017 13:24:28 -0800 Subject: does the underlying implementation (not public) of DatagramSocketImp's send receive method busy wait In-Reply-To: <585887490.4621806.1513274074018@mail.yahoo.com> References: <585887490.4621806.1513274074018.ref@mail.yahoo.com> <585887490.4621806.1513274074018@mail.yahoo.com> Message-ID: On Thu, Dec 14, 2017 at 9:54 AM, Andy Nuss wrote: > busy looping on a flag of the UDP protocol? > I believe it's just a blocking sendto(): https://github.com/netroby/jdk9-dev/blob/master/jdk/src/java.base/linux/native/libnet/linux_close.c#L393 So if the kernel decides to block, the thread will be suspended (consuming no CPU) until it's done. However, the UDP protocol has no concept of flow control, so blasting out packets without any rate limit or acknowledgement mechanism is not a good idea. If you saturate a link in the network, sendto() is not obligated to block, or provide any feedback whatsoever. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alan.Bateman at oracle.com Tue Dec 19 10:53:26 2017 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Tue, 19 Dec 2017 10:53:26 +0000 Subject: does the underlying implementation (not public) of DatagramSocketImp's send receive method busy wait In-Reply-To: References: <585887490.4621806.1513274074018.ref@mail.yahoo.com> <585887490.4621806.1513274074018@mail.yahoo.com> Message-ID: On 18/12/2017 21:24, Paul Marks wrote: > > > So if the kernel decides to block, the thread will be suspended > (consuming no CPU) until it's done. > > However, the UDP protocol has no concept of flow control, so blasting > out packets without any rate limit or acknowledgement mechanism is not > a good idea.? If you saturate a link in the network, sendto() is not > obligated to block, or provide any feedback whatsoever. That's right and you'll find that most kernels will just discard the packet so the send never blocks. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew_nuss at yahoo.com Tue Dec 19 13:59:42 2017 From: andrew_nuss at yahoo.com (Andy Nuss) Date: Tue, 19 Dec 2017 13:59:42 +0000 (UTC) Subject: does the underlying implementation (not public) of DatagramSocketImp's send receive method busy wait In-Reply-To: References: <585887490.4621806.1513274074018.ref@mail.yahoo.com> <585887490.4621806.1513274074018@mail.yahoo.com> Message-ID: <690659325.1477343.1513691982422@mail.yahoo.com> Aside from the fact that I should watch out for blasting too many packets at once, and realizing that this one "server/controller" machine has to control 100 or more client receiving machines, with smallish packets, is there any advantage at all relative to the linux kernel in having more than one UDP port open for sending out?? I.e. say the controller has a queue of 200 packets to spray out, is it sufficient to use just one java thread and one UDP outbound port, or would it give the kernel more throughput to send out the packets with 5 threads and 5 sockets or even more? The packets are 10 bytes and/or 140 bytes. On Tuesday, December 19, 2017, 2:53:30 AM PST, Alan Bateman wrote: On 18/12/2017 21:24, Paul Marks wrote: So if the kernel decides to block, the thread will be suspended (consuming no CPU) until it's done. However, the UDP protocol has no concept of flow control, so blasting out packets without any rate limit or acknowledgement mechanism is not a good idea.? If you saturate a link in the network, sendto() is not obligated to block, or provide any feedback whatsoever. That's right and you'll find that most kernels will just discard the packet so the send never blocks. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.x.mcmahon at oracle.com Tue Dec 19 14:20:53 2017 From: michael.x.mcmahon at oracle.com (Michael McMahon) Date: Tue, 19 Dec 2017 14:20:53 +0000 Subject: RFR: 8192966 HttpClient should reuse TCP connection for h2c connections [jdk10] In-Reply-To: <5A2FDA85.5070203@oracle.com> References: <5A2FAA5E.9000909@oracle.com> <6b207569-5587-c26c-c4db-d7ecb8600de1@oracle.com> <5A2FDA85.5070203@oracle.com> Message-ID: <5A392045.2090806@oracle.com> This is follow on from a previous review of the same bug fix. The fix is now targeted specifically for jdk10. A slightly different approach is taken this time. But, your comment below is incorporated into the new version Daniel Webrev: http://cr.openjdk.java.net/~michaelm/8192966/webrev.2/ Thanks Michael On 12/12/2017, 13:32, Michael McMahon wrote: > Thanks Daniel. I'm looking at this change again as I have > noticed the behavior when multiple https requests are initiated > might not be what we want. I'll post another webrev later. > > - Michael > > On 12/12/2017, 10:32, Daniel Fuchs wrote: >> Hi Michael, >> >> I wonder whether Http2Connection::closeStream would be >> a better place to call Stream::checkConnectionClosure? >> If not then shouldn't it be called in Stream::release >> as well? >> >> best regards, >> >> -- daniel >> >> On 12/12/2017 10:07, Michael McMahon wrote: >>> >>> http://cr.openjdk.java.net/~michaelm/8192966/webrev.1/ >>> >>> Thanks, >>> Michael. >> From daniel.fuchs at oracle.com Tue Dec 19 14:51:27 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Tue, 19 Dec 2017 14:51:27 +0000 Subject: RFR: 8192966 HttpClient should reuse TCP connection for h2c connections [jdk10] In-Reply-To: <5A392045.2090806@oracle.com> References: <5A2FAA5E.9000909@oracle.com> <6b207569-5587-c26c-c4db-d7ecb8600de1@oracle.com> <5A2FDA85.5070203@oracle.com> <5A392045.2090806@oracle.com> Message-ID: <18d76564-f21d-a003-7476-232c7af78f49@oracle.com> Hi Michael, On 19/12/2017 14:20, Michael McMahon wrote: > This is follow on from a previous review of the same bug fix. The fix > is now targeted specifically for jdk10. > > A slightly different approach is taken this time. But, your comment > below is incorporated into the new version Daniel > > Webrev: http://cr.openjdk.java.net/~michaelm/8192966/webrev.2/ This looks much better to me! best regards, -- daniel > > Thanks > Michael > > On 12/12/2017, 13:32, Michael McMahon wrote: >> Thanks Daniel. I'm looking at this change again as I have >> noticed the behavior when multiple https requests are initiated >> might not be what we want. I'll post another webrev later. >> >> - Michael >> >> On 12/12/2017, 10:32, Daniel Fuchs wrote: >>> Hi Michael, >>> >>> I wonder whether Http2Connection::closeStream would be >>> a better place to call Stream::checkConnectionClosure? >>> If not then shouldn't it be called in Stream::release >>> as well? >>> >>> best regards, >>> >>> -- daniel >>> >>> On 12/12/2017 10:07, Michael McMahon wrote: >>>> >>>> http://cr.openjdk.java.net/~michaelm/8192966/webrev.1/ >>>> >>>> Thanks, >>>> Michael. >>> From ecki at zusammenkunft.net Tue Dec 19 21:25:01 2017 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Tue, 19 Dec 2017 22:25:01 +0100 Subject: AW: does the underlying implementation (not public) ofDatagramSocketImp's send receive method busy wait In-Reply-To: <690659325.1477343.1513691982422@mail.yahoo.com> References: <585887490.4621806.1513274074018.ref@mail.yahoo.com> <585887490.4621806.1513274074018@mail.yahoo.com> <690659325.1477343.1513691982422@mail.yahoo.com> Message-ID: <5a3983ad.48d31c0a.b426e.9c3e@mx.google.com> Hello, it is a bit unlikely that there is much difference (especially for as low as 200 packets), but I would recommend you actually try it with your specific traffic pattern and System load. Not much a Java developer can say about various Linux kernel versions and their drivers. Gruss Bernd -- http://bernd.eckenfels.net Von: Andy Nuss Gesendet: Dienstag, 19. Dezember 2017 15:31 An: Alan Bateman; net-dev at openjdk.java.net; Paul Marks Betreff: Re: does the underlying implementation (not public) ofDatagramSocketImp's send receive method busy wait Aside from the fact that I should watch out for blasting too many packets at once, and realizing that this one "server/controller" machine has to control 100 or more client receiving machines, with smallish packets, is there any advantage at all relative to the linux kernel in having more than one UDP port open for sending out?? I.e. say the controller has a queue of 200 packets to spray out, is it sufficient to use just one java thread and one UDP outbound port, or would it give the kernel more throughput to send out the packets with 5 threads and 5 sockets or even more? The packets are 10 bytes and/or 140 bytes. On Tuesday, December 19, 2017, 2:53:30 AM PST, Alan Bateman wrote: On 18/12/2017 21:24, Paul Marks wrote: So if the kernel decides to block, the thread will be suspended (consuming no CPU) until it's done. However, the UDP protocol has no concept of flow control, so blasting out packets without any rate limit or acknowledgement mechanism is not a good idea.? If you saturate a link in the network, sendto() is not obligated to block, or provide any feedback whatsoever. That's right and you'll find that most kernels will just discard the packet so the send never blocks. -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.fuchs at oracle.com Wed Dec 20 09:29:32 2017 From: daniel.fuchs at oracle.com (Daniel Fuchs) Date: Wed, 20 Dec 2017 09:29:32 +0000 Subject: RFR 8193365: Improve interoperability between HTTP Client's BodyPublisher/BodySubscriber and Flow.Subscriber/Publisher In-Reply-To: <19cd5458-0851-6988-4d6e-d39f0b295396@oracle.com> References: <19cd5458-0851-6988-4d6e-d39f0b295396@oracle.com> Message-ID: <4a73cc9c-e1d3-7815-5c00-9f02e2931b35@oracle.com> Hi Chris, This looks good to me. I wonder if the implementations of Susbscriber in the tests should be strengthened. I see that onError or onNext could throw exception - and though SubscriberAdapter seems to (partly) cater for it in onError, it doesn't do anything for onNext. I also wonder if the wrapper (SubscriberAdapter) should forcefully cancel the subscribtion if its wrappee (the test Subscriber) throws unexpected errors or unchecked exception in onError/onNext/onComplete best regards, -- daniel For instance I see that On 15/12/2017 15:37, Chris Hegarty wrote: > This review is for 8193365 [1], to improve interoperability > between HTTP Client's BodyPublisher/BodySubscriber and > Flow.Subscriber/Publisher, that was raised as part of > feedback on JEP 321 [2]. > > http://cr.openjdk.java.net/~chegar/8193365/webrev.01/ > > -Chris. > > [1] https://bugs.openjdk.java.net/browse/JDK-8193365 > [2] > http://mail.openjdk.java.net/pipermail/net-dev/2017-December/thread.html#11063 > From chris.hegarty at oracle.com Wed Dec 20 15:01:35 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Wed, 20 Dec 2017 15:01:35 +0000 Subject: RFR 8193365: Improve interoperability between HTTP Client's BodyPublisher/BodySubscriber and Flow.Subscriber/Publisher In-Reply-To: <4a73cc9c-e1d3-7815-5c00-9f02e2931b35@oracle.com> References: <19cd5458-0851-6988-4d6e-d39f0b295396@oracle.com> <4a73cc9c-e1d3-7815-5c00-9f02e2931b35@oracle.com> Message-ID: <996dcd73-0d0a-054b-b8af-e59f937cf585@oracle.com> Thanks for the review Daniel. On 20/12/17 09:29, Daniel Fuchs wrote: > Hi Chris, > > This looks good to me. I wonder if the implementations of Susbscriber > in the tests should be strengthened. Yes, I think so. > I see that onError or onNext could throw exception - and > though SubscriberAdapter seems to (partly) cater for it in > onError, it doesn't do anything for onNext. Added a try/catch to handle this. > I also wonder if the wrapper (SubscriberAdapter) should forcefully > cancel the subscribtion if its wrappee (the test Subscriber) throws > unexpected errors or unchecked exception in onError/onNext/onComplete I think that would be best. Done. With some of these "thin" wrapper subscribers it's hard to know how much checking to perform, versus just pass-through to the downstream subscriber. I think the balance is right now. I also added additional cases for "too few" and "too many" bytes being published, as well as a few more NPE specific checks on the Publisher and Subscribers. Updated webrev: http://cr.openjdk.java.net/~chegar/8193365/webrev.02/ -Chris. > best regards, > > -- daniel > > For instance I see that > > On 15/12/2017 15:37, Chris Hegarty wrote: >> This review is for 8193365 [1], to improve interoperability >> between HTTP Client's BodyPublisher/BodySubscriber and >> Flow.Subscriber/Publisher, that was raised as part of >> feedback on JEP 321 [2]. >> >> http://cr.openjdk.java.net/~chegar/8193365/webrev.01/ >> >> -Chris. >> >> [1] https://bugs.openjdk.java.net/browse/JDK-8193365 >> [2] >> http://mail.openjdk.java.net/pipermail/net-dev/2017-December/thread.html#11063 >> > From david.holmes at oracle.com Fri Dec 22 01:27:14 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 22 Dec 2017 11:27:14 +1000 Subject: Adding SocketChannel toString to connection exception messages In-Reply-To: <160EE294-4802-4A5B-8EDE-1EB68EF329CE@gmail.com> References: <82A01DB9-7DB5-4D16-B890-FCB4306455DF@gmail.com> <160EE294-4802-4A5B-8EDE-1EB68EF329CE@gmail.com> Message-ID: cc'ing net-dev as that might be the more appropriate list. On 22/12/2017 10:59 AM, Steven Schlansker wrote: > >> On Dec 21, 2017, at 4:35 PM, David Holmes wrote: >> >> On 22/12/2017 10:29 AM, Steven Schlansker wrote: >>>> On Dec 21, 2017, at 11:11 AM, Steven Schlansker wrote: >>>> >>>> What if ConnectException included the attempted hostname / IP / port SocketAddress? >>>> java.net.ConnectException: Connection to 'foo.mycorp.com[10.x.x.x]:12345' refused >>>> Much more useful! This could also be extended to various other socket exceptions. >> >> I believe there are concerns with too much information that can be considered "sensitive" (like host names and IP addresses) appearing in error messages due to them ending up in log files and bug reports. > > Unfortunately that's exactly the information that is crucial to someone trying to diagnose issues... > Could it be an opt-in policy? Perhaps by a system property? The expectation is that such information should be added by layers higher in the call chain, rather than the JDK libraries assuming it is okay to do. > Currently the alternative I'm faced with is going through every piece of user code and library that *might* > throw this exception and wrapping it to add this critical diagnostic information. For an application that uses > java.net heavily, you can imagine how that is a tall task and possibly even not realistically achievable... > > (Is there a written policy regarding this somewhere, or is it up to the personal feelings of the contributors?) This is covered by the secure coding guidelines, under 'Confidential information': http://www.oracle.com/technetwork/java/seccodeguide-139067.html#2 "Guideline 2-1 / CONFIDENTIAL-1: Purge sensitive information from exceptions" I know for a fact that we'd have to scrub this information from test failures when putting the info into a bug report. If net-dev folk can't expand further on this then I suggest filing a CSR request so that the CSR group can consider it. But I think this will be a no-go (I'm a member of the CSR group). Cheers, David From chris.hegarty at oracle.com Fri Dec 22 09:57:48 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 22 Dec 2017 09:57:48 +0000 Subject: Adding SocketChannel toString to connection exception messages In-Reply-To: References: <82A01DB9-7DB5-4D16-B890-FCB4306455DF@gmail.com> <160EE294-4802-4A5B-8EDE-1EB68EF329CE@gmail.com> Message-ID: <3874615e-a801-67d9-b466-bdc70fa9e835@oracle.com> On 22/12/17 01:27, David Holmes wrote: > cc'ing net-dev as that might be the more appropriate list. > > On 22/12/2017 10:59 AM, Steven Schlansker wrote: >> >>> On Dec 21, 2017, at 4:35 PM, David Holmes >>> wrote: >>> >>> On 22/12/2017 10:29 AM, Steven Schlansker wrote: >>>>> On Dec 21, 2017, at 11:11 AM, Steven Schlansker >>>>> wrote: >>>>> >>>>> What if ConnectException included the attempted hostname / IP / >>>>> port SocketAddress? >>>>> java.net.ConnectException: Connection to >>>>> 'foo.mycorp.com[10.x.x.x]:12345' refused >>>>> Much more useful! This could also be extended to various other >>>>> socket exceptions. >>> >>> I believe there are concerns with too much information that can be >>> considered "sensitive" (like host names and IP addresses) appearing >>> in error messages due to them ending up in log files and bug reports. >> >> Unfortunately that's exactly the information that is crucial to >> someone trying to diagnose issues... And to someone trying to discern private information about a network. >> Could it be an opt-in policy? Perhaps by a system property? Well, you don't know where a log file or exception will end up. Most likely on stackoverflow. > The expectation is that such information should be added by layers > higher in the call chain, rather than the JDK libraries assuming it is > okay to do. Agreed. It may be some work, but if the issue is significant enough to the user then it may be worth their effort, as well as affording the opportunity to opt-out at any particular point in the code. >> Currently the alternative I'm faced with is going through every piece >> of user code and library that *might* >> throw this exception and wrapping it to add this critical diagnostic >> information. For an application that uses >> java.net heavily, you can imagine how that is a tall task and possibly >> even not realistically achievable... >> >> (Is there a written policy regarding this somewhere, or is it up to >> the personal feelings of the contributors?) > > This is covered by the secure coding guidelines, under 'Confidential > information': > > http://www.oracle.com/technetwork/java/seccodeguide-139067.html#2 > > "Guideline 2-1 / CONFIDENTIAL-1: Purge sensitive information from > exceptions" Exactly. > I know for a fact that we'd have to scrub this information from test > failures when putting the info into a bug report. > > If net-dev folk can't expand further on this then I suggest filing a CSR > request so that the CSR group can consider it. But I think this will be > a no-go (I'm a member of the CSR group). I have to agree with David here. I don't think that such a request could be supported. -Chris. From sean.coffey at oracle.com Fri Dec 22 14:59:42 2017 From: sean.coffey at oracle.com (=?UTF-8?Q?Se=c3=a1n_Coffey?=) Date: Fri, 22 Dec 2017 14:59:42 +0000 Subject: Adding SocketChannel toString to connection exception messages In-Reply-To: <3874615e-a801-67d9-b466-bdc70fa9e835@oracle.com> References: <82A01DB9-7DB5-4D16-B890-FCB4306455DF@gmail.com> <160EE294-4802-4A5B-8EDE-1EB68EF329CE@gmail.com> <3874615e-a801-67d9-b466-bdc70fa9e835@oracle.com> Message-ID: <1b83507a-bded-4e2b-1818-cd51e94d5176@oracle.com> As someone who works with alot of log files, I'd like to chime in and support Steven's end goal. Looking at a "Connection refused" error in the middle of a log file that possibly extends to millions of lines is near useless. In the era of cloud compute, diagnosing network issues is sure to become a more common task. While we may not be able to put the sensitive information in an exception message, I think we could put it behind a (new?) system property which might be able to log this information. Logs contain all sorts of sensitive data. Take javax.net.debug=ssl output for example. Regards, Sean. On 22/12/17 09:57, Chris Hegarty wrote: > On 22/12/17 01:27, David Holmes wrote: >> cc'ing net-dev as that might be the more appropriate list. >> >> On 22/12/2017 10:59 AM, Steven Schlansker wrote: >>> >>>> On Dec 21, 2017, at 4:35 PM, David Holmes >>>> wrote: >>>> >>>> On 22/12/2017 10:29 AM, Steven Schlansker wrote: >>>>>> On Dec 21, 2017, at 11:11 AM, Steven Schlansker >>>>>> wrote: >>>>>> >>>>>> What if ConnectException included the attempted hostname / IP / >>>>>> port SocketAddress? >>>>>> java.net.ConnectException: Connection to >>>>>> 'foo.mycorp.com[10.x.x.x]:12345' refused >>>>>> Much more useful! This could also be extended to various other >>>>>> socket exceptions. >>>> >>>> I believe there are concerns with too much information that can be >>>> considered "sensitive" (like host names and IP addresses) appearing >>>> in error messages due to them ending up in log files and bug reports. >>> >>> Unfortunately that's exactly the information that is crucial to >>> someone trying to diagnose issues... > > And to someone trying to discern private information about a network. > >>> Could it be an opt-in policy? Perhaps by a system property? > > Well, you don't know where a log file or exception will end up. > Most likely on stackoverflow. > >> The expectation is that such information should be added by layers >> higher in the call chain, rather than the JDK libraries assuming it >> is okay to do. > > Agreed. It may be some work, but if the issue is significant > enough to the user then it may be worth their effort, as well > as affording the opportunity to opt-out at any particular > point in the code. > >>> Currently the alternative I'm faced with is going through every >>> piece of user code and library that *might* >>> throw this exception and wrapping it to add this critical diagnostic >>> information. For an application that uses >>> java.net heavily, you can imagine how that is a tall task and >>> possibly even not realistically achievable... >>> >>> (Is there a written policy regarding this somewhere, or is it up to >>> the personal feelings of the contributors?) >> >> This is covered by the secure coding guidelines, under 'Confidential >> information': >> >> http://www.oracle.com/technetwork/java/seccodeguide-139067.html#2 >> >> "Guideline 2-1 / CONFIDENTIAL-1: Purge sensitive information from >> exceptions" > > Exactly. > >> I know for a fact that we'd have to scrub this information from test >> failures when putting the info into a bug report. >> >> If net-dev folk can't expand further on this then I suggest filing a >> CSR request so that the CSR group can consider it. But I think this >> will be a no-go (I'm a member of the CSR group). > > I have to agree with David here. I don't think that such a request > could be supported. > > -Chris. From chris.hegarty at oracle.com Fri Dec 22 15:17:31 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 22 Dec 2017 15:17:31 +0000 Subject: Adding SocketChannel toString to connection exception messages In-Reply-To: <1b83507a-bded-4e2b-1818-cd51e94d5176@oracle.com> References: <82A01DB9-7DB5-4D16-B890-FCB4306455DF@gmail.com> <160EE294-4802-4A5B-8EDE-1EB68EF329CE@gmail.com> <3874615e-a801-67d9-b466-bdc70fa9e835@oracle.com> <1b83507a-bded-4e2b-1818-cd51e94d5176@oracle.com> Message-ID: <8546d3d3-dd76-059c-dd28-03ca2f782c01@oracle.com> On 22/12/17 14:59, Se?n Coffey wrote: > As someone who works with alot of log files, I'd like to chime in and > support Steven's end goal. Looking at a "Connection refused" error in > the middle of a log file that possibly extends to millions of lines is > near useless. In the era of cloud compute, diagnosing network issues is > sure to become a more common task. > > While we may not be able to put the sensitive information in an > exception message, I think we could put it behind a (new?) system > property which might be able to log this information. Logs contain all > sorts of sensitive data. Take javax.net.debug=ssl output for example. I have some sympathy for (capital-L)ogging such detail messages ( given the reasonable restriction on access to log files ), but a system property that effectively amends exception detail messages, or prints to stdout/stderr is not a runner in my opinion. Maybe we should be looking at instrumentation with JFR events, or similar. My point being, if someone has time and energy enough to spend on this, then we can do better than javax.net.debug=ssl. Also, someone should check that divulging such sensitive information, even in log files, is acceptable from a security perspective. I'm personally still dubious. -Chris. From ecki at zusammenkunft.net Fri Dec 22 16:42:45 2017 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 22 Dec 2017 16:42:45 +0000 Subject: Adding SocketChannel toString to connection exception messages In-Reply-To: <8546d3d3-dd76-059c-dd28-03ca2f782c01@oracle.com> References: <82A01DB9-7DB5-4D16-B890-FCB4306455DF@gmail.com> <160EE294-4802-4A5B-8EDE-1EB68EF329CE@gmail.com> <3874615e-a801-67d9-b466-bdc70fa9e835@oracle.com> <1b83507a-bded-4e2b-1818-cd51e94d5176@oracle.com>, <8546d3d3-dd76-059c-dd28-03ca2f782c01@oracle.com> Message-ID: Hello, I also dearly miss Socket addresses in connection exceptions, however it looks like it is not going to make it. However if we add a getter for the Peer address (not included in toString) then logging frameworks could detect instances of ConnectException and process them accordingly. Gruss Bernd -- http://bernd.eckenfels.net ________________________________ From: net-dev on behalf of Chris Hegarty Sent: Friday, December 22, 2017 4:17:31 PM To: Se?n Coffey; David Holmes; Steven Schlansker Cc: core-libs-dev; net-dev at openjdk.java.net Subject: Re: Adding SocketChannel toString to connection exception messages On 22/12/17 14:59, Se?n Coffey wrote: > As someone who works with alot of log files, I'd like to chime in and > support Steven's end goal. Looking at a "Connection refused" error in > the middle of a log file that possibly extends to millions of lines is > near useless. In the era of cloud compute, diagnosing network issues is > sure to become a more common task. > > While we may not be able to put the sensitive information in an > exception message, I think we could put it behind a (new?) system > property which might be able to log this information. Logs contain all > sorts of sensitive data. Take javax.net.debug=ssl output for example. I have some sympathy for (capital-L)ogging such detail messages ( given the reasonable restriction on access to log files ), but a system property that effectively amends exception detail messages, or prints to stdout/stderr is not a runner in my opinion. Maybe we should be looking at instrumentation with JFR events, or similar. My point being, if someone has time and energy enough to spend on this, then we can do better than javax.net.debug=ssl. Also, someone should check that divulging such sensitive information, even in log files, is acceptable from a security perspective. I'm personally still dubious. -Chris. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.hegarty at oracle.com Fri Dec 29 17:15:14 2017 From: chris.hegarty at oracle.com (Chris Hegarty) Date: Fri, 29 Dec 2017 17:15:14 +0000 Subject: Adding SocketChannel toString to connection exception messages In-Reply-To: References: <82A01DB9-7DB5-4D16-B890-FCB4306455DF@gmail.com> <160EE294-4802-4A5B-8EDE-1EB68EF329CE@gmail.com> <3874615e-a801-67d9-b466-bdc70fa9e835@oracle.com> <1b83507a-bded-4e2b-1818-cd51e94d5176@oracle.com> <8546d3d3-dd76-059c-dd28-03ca2f782c01@oracle.com> Message-ID: > On 29 Dec 2017, at 00:33, Steven Schlansker wrote: > > Thanks for the discussion! > > So, it sounds like amending the message by default is going to be a non-starter -- but at least adding the information otherwise might be possible. > > One possibility would be to add new fields to SocketException, for example an InetSocketAddress representing both the local and remote (if available). You would need to careful to not disclose resolved addresses to untrusted code. SocketException, since a subtype of IOException, can wind up in many places. Would you be proposing to add these new fields to the serial-form of SocketException? What behaviour do you envisage for deserializing instances from previous releases? This will have an impact of any potential API. > This would not be rendered by default, but could be retrieved by a cooperating application or library. Or maybe a second step could be a '-Djavax.net.debug=exceptions' that then appends this information by default, but that sounds more controversial. Logging seems like a better alternative than a system property, to me. > Then at least libraries and applications have the ability to get these diagnostics, even if they choose not to. > Is this an acceptable approach? Would it be accepted as a patch? I suspect that a webrev/patch would help drive the discussion, rather than a commitment to accepting it into the JDK. -Chris. >> On Dec 22, 2017, at 8:42 AM, Bernd Eckenfels wrote: >> >> Hello, >> >> I also dearly miss Socket addresses in connection exceptions, however it looks like it is not going to make it. However if we add a getter for the Peer address (not included in toString) then logging frameworks could detect instances of ConnectException and process them accordingly. >> >> Gruss >> Bernd >> -- >> http://bernd.eckenfels.net >> ________________________________ >> From: net-dev on behalf of Chris Hegarty >> Sent: Friday, December 22, 2017 4:17:31 PM >> To: Se?n Coffey; David Holmes; Steven Schlansker >> Cc: core-libs-dev; net-dev at openjdk.java.net >> Subject: Re: Adding SocketChannel toString to connection exception messages >> >> >>> On 22/12/17 14:59, Se?n Coffey wrote: >>> As someone who works with alot of log files, I'd like to chime in and >>> support Steven's end goal. Looking at a "Connection refused" error in >>> the middle of a log file that possibly extends to millions of lines is >>> near useless. In the era of cloud compute, diagnosing network issues is >>> sure to become a more common task. >>> >>> While we may not be able to put the sensitive information in an >>> exception message, I think we could put it behind a (new?) system >>> property which might be able to log this information. Logs contain all >>> sorts of sensitive data. Take javax.net.debug=ssl output for example. >> >> I have some sympathy for (capital-L)ogging such detail messages >> ( given the reasonable restriction on access to log files ), but >> a system property that effectively amends exception detail >> messages, or prints to stdout/stderr is not a runner in my >> opinion. >> >> Maybe we should be looking at instrumentation with JFR events, or >> similar. My point being, if someone has time and energy enough >> to spend on this, then we can do better than javax.net.debug=ssl. >> Also, someone should check that divulging such sensitive information, >> even in log files, is acceptable from a security perspective. I'm >> personally still dubious. >> >> -Chris. >