From ivan.gerasimov at oracle.com Mon Oct 5 13:08:21 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 5 Oct 2015 16:08:21 +0300 Subject: [9] [XS] RFR 8138819 : File descriptors leak during construction of selector Message-ID: <56127645.7010206@oracle.com> Hello! When constructing different kinds of Selector, a pair of file descriptors (the pipe ends) are created. If later in the constructor an exception is thrown, these will remain open. Would you please help review the fix? BUGURL: https://bugs.openjdk.java.net/browse/JDK-8138819 WEBREV: http://cr.openjdk.java.net/~igerasim/8138819/00/webrev/ Sincerely yours, Ivan From Roger.Riggs at Oracle.com Mon Oct 5 13:12:14 2015 From: Roger.Riggs at Oracle.com (Roger Riggs) Date: Mon, 5 Oct 2015 09:12:14 -0400 Subject: [9] [XS] RFR 8138819 : File descriptors leak during construction of selector In-Reply-To: <56127645.7010206@oracle.com> References: <56127645.7010206@oracle.com> Message-ID: <5612772E.4020604@Oracle.com> Hi Ivan, Looks fine, Roger (not specifically an NIO reviewer) On 10/5/2015 9:08 AM, Ivan Gerasimov wrote: > Hello! > > When constructing different kinds of Selector, a pair of file > descriptors (the pipe ends) are created. > If later in the constructor an exception is thrown, these will remain > open. > Would you please help review the fix? > > BUGURL: https://bugs.openjdk.java.net/browse/JDK-8138819 > WEBREV: http://cr.openjdk.java.net/~igerasim/8138819/00/webrev/ > > Sincerely yours, > Ivan From ivan.gerasimov at oracle.com Mon Oct 5 13:18:38 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 5 Oct 2015 16:18:38 +0300 Subject: [9] [XS] RFR 8138819 : File descriptors leak during construction of selector In-Reply-To: <5612772E.4020604@Oracle.com> References: <56127645.7010206@oracle.com> <5612772E.4020604@Oracle.com> Message-ID: <561278AE.7020003@oracle.com> Thanks Roger! On 05.10.2015 16:12, Roger Riggs wrote: > Hi Ivan, > > Looks fine, Roger > > (not specifically an NIO reviewer) > I'll wait 24 hours in a case anyone would like to comment on the suggested fix. Sincerely yours, Ivan > On 10/5/2015 9:08 AM, Ivan Gerasimov wrote: >> Hello! >> >> When constructing different kinds of Selector, a pair of file >> descriptors (the pipe ends) are created. >> If later in the constructor an exception is thrown, these will remain >> open. >> Would you please help review the fix? >> >> BUGURL: https://bugs.openjdk.java.net/browse/JDK-8138819 >> WEBREV: http://cr.openjdk.java.net/~igerasim/8138819/00/webrev/ >> >> Sincerely yours, >> Ivan > > From Alan.Bateman at oracle.com Mon Oct 5 13:24:53 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Mon, 5 Oct 2015 14:24:53 +0100 Subject: [9] [XS] RFR 8138819 : File descriptors leak during construction of selector In-Reply-To: <56127645.7010206@oracle.com> References: <56127645.7010206@oracle.com> Message-ID: <56127A25.7050907@oracle.com> On 05/10/2015 14:08, Ivan Gerasimov wrote: > Hello! > > When constructing different kinds of Selector, a pair of file > descriptors (the pipe ends) are created. > If later in the constructor an exception is thrown, these will remain > open. > Would you please help review the fix? > > BUGURL: https://bugs.openjdk.java.net/browse/JDK-8138819 > WEBREV: http://cr.openjdk.java.net/~igerasim/8138819/00/webrev/ (I've just changed the synopsis on this bug as the original synopsis was misleading. Can you use the new synopsis for the change-set message?). The patch looks okay although if OOME is thrown then I assume the addSuppressed might fail. I was going to suggest we create the wrapper before the socket pair so that the rollback is one close rather than two but it's probably not worth it. -Alan. From ivan.gerasimov at oracle.com Mon Oct 5 13:40:50 2015 From: ivan.gerasimov at oracle.com (Ivan Gerasimov) Date: Mon, 5 Oct 2015 16:40:50 +0300 Subject: [9] [XS] RFR 8138819 : File descriptors leak during construction of selector In-Reply-To: <56127A25.7050907@oracle.com> References: <56127645.7010206@oracle.com> <56127A25.7050907@oracle.com> Message-ID: <56127DE2.1080202@oracle.com> Thanks Alan! On 05.10.2015 16:24, Alan Bateman wrote: > > On 05/10/2015 14:08, Ivan Gerasimov wrote: >> Hello! >> >> When constructing different kinds of Selector, a pair of file >> descriptors (the pipe ends) are created. >> If later in the constructor an exception is thrown, these will remain >> open. >> Would you please help review the fix? >> >> BUGURL: https://bugs.openjdk.java.net/browse/JDK-8138819 >> WEBREV: http://cr.openjdk.java.net/~igerasim/8138819/00/webrev/ > > (I've just changed the synopsis on this bug as the original synopsis > was misleading. Can you use the new synopsis for the change-set > message?). > Yes, sure. Thank your for correcting it. > The patch looks okay although if OOME is thrown then I assume the > addSuppressed might fail. > It that case one OOME will be replaces with another with a slightly different stack trace. As this situation would be extremely rare, I'm not sure it worth complicating the code. Sincerely yours, Ivan > I was going to suggest we create the wrapper before the socket pair so > that the rollback is one close rather than two but it's probably not > worth it. > > -Alan. > > > From brian.burkhalter at oracle.com Wed Oct 21 01:05:09 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Tue, 20 Oct 2015 18:05:09 -0700 Subject: [9] RFR of 8139133: Changing the modification time on a unix domain socket file fails Message-ID: <9910F845-0B46-4446-8320-B1BFBA19E017@oracle.com> Please review at your convenience. Issue: https://bugs.openjdk.java.net/browse/JDK-8139133 Patch: http://cr.openjdk.java.net/~bpb/8139133/webrev.00/ The exception is caused by the native open(2) system call's failing with errno ENXIO, "No such device or address.? The fix is to use equivalent methods which do not depend on the file descriptor?s being available. Thanks, Brian From Alan.Bateman at oracle.com Wed Oct 21 01:27:57 2015 From: Alan.Bateman at oracle.com (Alan Bateman) Date: Wed, 21 Oct 2015 02:27:57 +0100 Subject: [9] RFR of 8139133: Changing the modification time on a unix domain socket file fails In-Reply-To: <9910F845-0B46-4446-8320-B1BFBA19E017@oracle.com> References: <9910F845-0B46-4446-8320-B1BFBA19E017@oracle.com> Message-ID: <5626EA1D.1070005@oracle.com> On 21/10/2015 02:05, Brian Burkhalter wrote: > Please review at your convenience. > > Issue: https://bugs.openjdk.java.net/browse/JDK-8139133 > Patch: http://cr.openjdk.java.net/~bpb/8139133/webrev.00/ > > The exception is caused by the native open(2) system call's failing with errno ENXIO, "No such device or address.? The fix is to use equivalent methods which do not depend on the file descriptor?s being available. > Catching IOException doesn't look right. Instead, I think this will need re-visiting openForAttributeAccess so that you can examine the errno and only fallback to the file path for ENXIO and maybe a few other errors. A minor comment is renaming isFdValid to haveFd could make the usages a bit clearer. Does the test have a race? It looks like it might attempt to accept the socket file before the nc command creates it. Also can the test be moved to attribute/BasicFileAttributeView as that is where the tests for setTimes are (it's not a test for the FileTime class). -Alan. From brian.burkhalter at oracle.com Wed Oct 21 15:16:03 2015 From: brian.burkhalter at oracle.com (Brian Burkhalter) Date: Wed, 21 Oct 2015 08:16:03 -0700 Subject: [9] RFR of 8139133: Changing the modification time on a unix domain socket file fails In-Reply-To: <5626EA1D.1070005@oracle.com> References: <9910F845-0B46-4446-8320-B1BFBA19E017@oracle.com> <5626EA1D.1070005@oracle.com> Message-ID: On Oct 20, 2015, at 6:27 PM, Alan Bateman wrote: > On 21/10/2015 02:05, Brian Burkhalter wrote: >> Please review at your convenience. >> >> Issue: https://bugs.openjdk.java.net/browse/JDK-8139133 >> Patch: http://cr.openjdk.java.net/~bpb/8139133/webrev.00/ >> >> The exception is caused by the native open(2) system call's failing with errno ENXIO, "No such device or address.? The fix is to use equivalent methods which do not depend on the file descriptor?s being available. >> > Catching IOException doesn't look right. Instead, I think this will need re-visiting openForAttributeAccess so that you can examine the errno and only fallback to the file path for ENXIO and maybe a few other errors. That was more or less what I intended to do in the first place but opted for a simpler change. I think your suggestion is safer however. > A minor comment is renaming isFdValid to haveFd could make the usages a bit clearer. > > Does the test have a race? It looks like it might attempt to accept the socket file before the nc command creates it. I?ll investigate. > Also can the test be moved to attribute/BasicFileAttributeView as that is where the tests for setTimes are (it's not a test for the FileTime class). OK. Thanks, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.bergamin at wedgenetworks.com Thu Oct 22 20:50:40 2015 From: patrick.bergamin at wedgenetworks.com (Patrick Bergamin) Date: Thu, 22 Oct 2015 14:50:40 -0600 Subject: Memory usage of EPollArraySelector Message-ID: <56294C20.4030300@wedgenetworks.com> I'm having problems with memory usage of the current implementation of EPollArraySelector in 1.8.0_60 for an existing proxy application. We've been on java version 1.7.0_05 for a while now because the new implementation of EPollArraySelector does not work well with the design of this proxy application. I did find the sun.nio.ch.maxUpdateArraySize property helped to reduce memory usage a bit. But as the proxy application runs all the EPollArraySelector objects will slowly accumulate memory. The current design of the proxy application is to have one Selector handle the listen socket. Once a connection is accepted, management of the connection is handed off to another thread on another Selector. Basically there is one Selector allocated per socket connection. The Selectors are never closed they are reused when a new socket connection is accepted. The application was designed before this latest implementation of EPollArraySelector. Redesigning the application to decouple the Selector from the socket connection would be a fair amount work. We have machines that are capable of running this proxy application with around 350,000 open connections. Since it is a proxy there are actually 700,000 open connections. The file descriptor limit is set high (5,000,000) to be able to handle all these open socket connections. Below I've included a patch to show what kinds of things I need to do to bring the memory usage of EPollArraySelector down for this proxy application. Is there any interest in including a second epoll implementation in openjdk that uses less memory or perhaps have more properties to control the memory usage of the existing EPollArraySelector? thanks, Patrick Bergamin --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 -0600 @@ -29,6 +29,7 @@ import java.security.AccessController; import java.util.BitSet; import java.util.HashMap; +import java.util.HashSet; import java.util.Map; import sun.security.action.GetIntegerAction; @@ -122,7 +123,7 @@ // Used by release and updateRegistrations to track whether a file // descriptor is registered with epoll. - private final BitSet registered = new BitSet(); + private final HashSet registered = new HashSet(); EPollArrayWrapper() throws IOException { @@ -187,7 +188,10 @@ } } else { Integer key = Integer.valueOf(fd); - if (!isEventsHighKilled(key) || force) { + if (events == KILLED) { + eventsHigh.remove(key); + } + else if (!isEventsHighKilled(key) || force) { eventsHigh.put(key, Byte.valueOf(events)); } } @@ -201,6 +205,9 @@ return eventsLow[fd]; } else { Byte result = eventsHigh.get(Integer.valueOf(fd)); + if (result == null) { + return KILLED; + } // result should never be null return result.byteValue(); } @@ -235,7 +242,7 @@ // force the initial update events to 0 as it may be KILLED by a // previous registration. synchronized (updateLock) { - assert !registered.get(fd); + assert !registered.contains(fd); setUpdateEvents(fd, (byte)0, true); } } @@ -249,9 +256,9 @@ setUpdateEvents(fd, KILLED, false); // remove from epoll - if (registered.get(fd)) { + if (registered.contains(fd)) { epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); - registered.clear(fd); + registered.remove(fd); } } } @@ -286,7 +293,7 @@ while (j < updateCount) { int fd = updateDescriptors[j]; short events = getUpdateEvents(fd); - boolean isRegistered = registered.get(fd); + boolean isRegistered = registered.contains(fd); int opcode = 0; if (events != KILLED) { @@ -298,9 +305,9 @@ if (opcode != 0) { epollCtl(epfd, opcode, fd, events); if (opcode == EPOLL_CTL_ADD) { - registered.set(fd); + registered.add(fd); } else if (opcode == EPOLL_CTL_DEL) { - registered.clear(fd); + registered.remove(fd); } } } From vitalyd at gmail.com Thu Oct 22 21:39:35 2015 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Thu, 22 Oct 2015 17:39:35 -0400 Subject: Memory usage of EPollArraySelector In-Reply-To: <56294C20.4030300@wedgenetworks.com> References: <56294C20.4030300@wedgenetworks.com> Message-ID: Patrick, I'm confused - how many selectors do you have? You mentioned a selector is created for each accepted connection but then state that selectors are reused for newly accepted connections. Do you mean a selector is reused after its sole connection is closed? You should bite the bullet and reimplement your application. If I'm reading right, you're wasting lots of resources all around, and your proposal is just a hack (with adverse performance due to additional memory chasing via HashSet) that will likely catch up to you anyway. sent from my phone On Oct 22, 2015 4:51 PM, "Patrick Bergamin" < patrick.bergamin at wedgenetworks.com> wrote: > I'm having problems with memory usage of the current implementation > of EPollArraySelector in 1.8.0_60 for an existing proxy application. > We've been on java version 1.7.0_05 for a while now because the new > implementation of EPollArraySelector does not work well with the > design of this proxy application. > > I did find the sun.nio.ch.maxUpdateArraySize property helped to > reduce memory usage a bit. But as the proxy application runs > all the EPollArraySelector objects will slowly accumulate memory. > > The current design of the proxy application is to have one Selector > handle the listen socket. Once a connection is accepted, management > of the connection is handed off to another thread on another Selector. > Basically there is one Selector allocated per socket connection. > The Selectors are never closed they are reused when a new socket > connection is accepted. > > The application was designed before this latest implementation of > EPollArraySelector. Redesigning the application to decouple the > Selector from the socket connection would be a fair amount work. > > We have machines that are capable of running this proxy application > with around 350,000 open connections. Since it is a proxy there > are actually 700,000 open connections. The file descriptor limit > is set high (5,000,000) to be able to handle all these open socket > connections. > > Below I've included a patch to show what kinds of things I need > to do to bring the memory usage of EPollArraySelector down for > this proxy application. > > Is there any interest in including a second epoll implementation > in openjdk that uses less memory or perhaps have more properties > to control the memory usage of the existing EPollArraySelector? > > thanks, > Patrick Bergamin > > --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 > +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 -0600 > @@ -29,6 +29,7 @@ > import java.security.AccessController; > import java.util.BitSet; > import java.util.HashMap; > +import java.util.HashSet; > import java.util.Map; > import sun.security.action.GetIntegerAction; > > @@ -122,7 +123,7 @@ > > // Used by release and updateRegistrations to track whether a file > // descriptor is registered with epoll. > - private final BitSet registered = new BitSet(); > + private final HashSet registered = new HashSet(); > > > EPollArrayWrapper() throws IOException { > @@ -187,7 +188,10 @@ > } > } else { > Integer key = Integer.valueOf(fd); > - if (!isEventsHighKilled(key) || force) { > + if (events == KILLED) { > + eventsHigh.remove(key); > + } > + else if (!isEventsHighKilled(key) || force) { > eventsHigh.put(key, Byte.valueOf(events)); > } > } > @@ -201,6 +205,9 @@ > return eventsLow[fd]; > } else { > Byte result = eventsHigh.get(Integer.valueOf(fd)); > + if (result == null) { > + return KILLED; > + } > // result should never be null > return result.byteValue(); > } > @@ -235,7 +242,7 @@ > // force the initial update events to 0 as it may be KILLED by a > // previous registration. > synchronized (updateLock) { > - assert !registered.get(fd); > + assert !registered.contains(fd); > setUpdateEvents(fd, (byte)0, true); > } > } > @@ -249,9 +256,9 @@ > setUpdateEvents(fd, KILLED, false); > > // remove from epoll > - if (registered.get(fd)) { > + if (registered.contains(fd)) { > epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); > - registered.clear(fd); > + registered.remove(fd); > } > } > } > @@ -286,7 +293,7 @@ > while (j < updateCount) { > int fd = updateDescriptors[j]; > short events = getUpdateEvents(fd); > - boolean isRegistered = registered.get(fd); > + boolean isRegistered = registered.contains(fd); > int opcode = 0; > > if (events != KILLED) { > @@ -298,9 +305,9 @@ > if (opcode != 0) { > epollCtl(epfd, opcode, fd, events); > if (opcode == EPOLL_CTL_ADD) { > - registered.set(fd); > + registered.add(fd); > } else if (opcode == EPOLL_CTL_DEL) { > - registered.clear(fd); > + registered.remove(fd); > } > } > } > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From weijun.wang at oracle.com Fri Oct 23 09:38:22 2015 From: weijun.wang at oracle.com (Weijun Wang) Date: Fri, 23 Oct 2015 17:38:22 +0800 Subject: Strange "..\" on Windows Message-ID: <562A000E.1010908@oracle.com> Paths.get("x").relativize(Paths.get("")) returns ..\ on Windows, which is not the same as Paths.get("..\\") (which shows as ..). In fact, its getName(0).toString() is exactly ..\. It looks like the \ is not treated as a path separator. Also, Paths.get("x/y").relativize(Paths.get("")) returns ..\..\ on Windows, but its getName(0) is .. and getName(1) is ..\ On Mac, it's all normal. Is this a bug? Thanks Max p.s. I run in cygwin of Windows 10. From patrick.bergamin at wedgenetworks.com Fri Oct 23 16:06:35 2015 From: patrick.bergamin at wedgenetworks.com (Patrick Bergamin) Date: Fri, 23 Oct 2015 10:06:35 -0600 Subject: Memory usage of EPollArraySelector In-Reply-To: References: <56294C20.4030300@wedgenetworks.com> Message-ID: <562A5B0B.4080708@wedgenetworks.com> On 15-10-22 03:39 PM, Vitaly Davidovich wrote: > > Patrick, > Yes Selectors are reused. One Selector is used per connection. When a socket connection is closed the Selectors are cached along with a connection handler object so they can be reused when new socket connections are accepted. > I'm confused - how many selectors do you have? You mentioned a > selector is created for each accepted connection but then state that > selectors are reused for newly accepted connections. Do you mean a > selector is reused after its sole connection is closed? > I'm not sure what you mean by memory chasing. I changed the BitSet to a HashSet because the file descriptor numbers can get easily get into the millions. BitSet can allocate a good chunk of memory even if only a couple of file descriptors are registered with it (if the file descriptor numbers are large). When you have hundreds of thousands of open Selectors the memory usage adds up. This is one reason why simply closing Selectors after a connection is closed does not solve the memory usage problem. From my perspective it is the EpollArrayWrapper that is wasting memory resources. It can allocate memory that is not used and never releases any that is allocated. Likely this was done for performance reasons. I haven't looked back through the revision history. I'm not proposing the suggested diff to be committed but rather I'm presenting it to show where the memory is accumulating in the application. After sun.nio.ch.maxUpdateArraySize was set to a smaller value it was the 'registered' object and 'eventsHigh' object within EpollArrayWrapper that were accumulating memory. I'm asking if there is something that can be done to reduce the memory usage of EpollArrayWrapper by either adding more system properties to change its behaviour or create a second implementation? thanks, Patrick Bergamin > You should bite the bullet and reimplement your application. If I'm > reading right, you're wasting lots of resources all around, and your > proposal is just a hack (with adverse performance due to additional > memory chasing via HashSet) that will likely catch up to you anyway. > > sent from my phone > > On Oct 22, 2015 4:51 PM, "Patrick Bergamin" > > wrote: > > I'm having problems with memory usage of the current implementation > of EPollArraySelector in 1.8.0_60 for an existing proxy application. > We've been on java version 1.7.0_05 for a while now because the new > implementation of EPollArraySelector does not work well with the > design of this proxy application. > > I did find the sun.nio.ch.maxUpdateArraySize property helped to > reduce memory usage a bit. But as the proxy application runs > all the EPollArraySelector objects will slowly accumulate memory. > > The current design of the proxy application is to have one Selector > handle the listen socket. Once a connection is accepted, management > of the connection is handed off to another thread on another Selector. > Basically there is one Selector allocated per socket connection. > The Selectors are never closed they are reused when a new socket > connection is accepted. > > The application was designed before this latest implementation of > EPollArraySelector. Redesigning the application to decouple the > Selector from the socket connection would be a fair amount work. > > We have machines that are capable of running this proxy application > with around 350,000 open connections. Since it is a proxy there > are actually 700,000 open connections. The file descriptor limit > is set high (5,000,000) to be able to handle all these open socket > connections. > > Below I've included a patch to show what kinds of things I need > to do to bring the memory usage of EPollArraySelector down for > this proxy application. > > Is there any interest in including a second epoll implementation > in openjdk that uses less memory or perhaps have more properties > to control the memory usage of the existing EPollArraySelector? > > thanks, > Patrick Bergamin > > --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 > +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 -0600 > @@ -29,6 +29,7 @@ > import java.security.AccessController; > import java.util.BitSet; > import java.util.HashMap; > +import java.util.HashSet; > import java.util.Map; > import sun.security.action.GetIntegerAction; > > @@ -122,7 +123,7 @@ > > // Used by release and updateRegistrations to track whether a > file > // descriptor is registered with epoll. > - private final BitSet registered = new BitSet(); > + private final HashSet registered = new > HashSet(); > > > EPollArrayWrapper() throws IOException { > @@ -187,7 +188,10 @@ > } > } else { > Integer key = Integer.valueOf(fd); > - if (!isEventsHighKilled(key) || force) { > + if (events == KILLED) { > + eventsHigh.remove(key); > + } > + else if (!isEventsHighKilled(key) || force) { > eventsHigh.put(key, Byte.valueOf(events)); > } > } > @@ -201,6 +205,9 @@ > return eventsLow[fd]; > } else { > Byte result = eventsHigh.get(Integer.valueOf(fd)); > + if (result == null) { > + return KILLED; > + } > // result should never be null > return result.byteValue(); > } > @@ -235,7 +242,7 @@ > // force the initial update events to 0 as it may be > KILLED by a > // previous registration. > synchronized (updateLock) { > - assert !registered.get(fd); > + assert !registered.contains(fd); > setUpdateEvents(fd, (byte)0, true); > } > } > @@ -249,9 +256,9 @@ > setUpdateEvents(fd, KILLED, false); > > // remove from epoll > - if (registered.get(fd)) { > + if (registered.contains(fd)) { > epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); > - registered.clear(fd); > + registered.remove(fd); > } > } > } > @@ -286,7 +293,7 @@ > while (j < updateCount) { > int fd = updateDescriptors[j]; > short events = getUpdateEvents(fd); > - boolean isRegistered = registered.get(fd); > + boolean isRegistered = registered.contains(fd); > int opcode = 0; > > if (events != KILLED) { > @@ -298,9 +305,9 @@ > if (opcode != 0) { > epollCtl(epfd, opcode, fd, events); > if (opcode == EPOLL_CTL_ADD) { > - registered.set(fd); > + registered.add(fd); > } else if (opcode == EPOLL_CTL_DEL) { > - registered.clear(fd); > + registered.remove(fd); > } > } > } > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitalyd at gmail.com Fri Oct 23 16:23:01 2015 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Fri, 23 Oct 2015 12:23:01 -0400 Subject: Memory usage of EPollArraySelector In-Reply-To: <562A5B0B.4080708@wedgenetworks.com> References: <56294C20.4030300@wedgenetworks.com> <562A5B0B.4080708@wedgenetworks.com> Message-ID: The entire problem is you have a Selector per connection rather than Selector per worker thread, as I mentioned in my previous reply. I don't think Selector was designed for such a thing, and so using a BitSet makes sense for the intended usage. HashSet will reduce your memory footprint because you'll have just 1 entry in there, but the common/intended case will make it worse as it has terrible memory locality properties. Your real fix is to redesign the Selector usage within your application. On Fri, Oct 23, 2015 at 12:06 PM, Patrick Bergamin < patrick.bergamin at wedgenetworks.com> wrote: > On 15-10-22 03:39 PM, Vitaly Davidovich wrote: > > Patrick, > > > Yes Selectors are reused. One Selector is used per connection. When a > socket connection is closed the Selectors are cached along with a > connection handler object so they can be reused when new socket connections > are accepted. > > I'm confused - how many selectors do you have? You mentioned a selector is > created for each accepted connection but then state that selectors are > reused for newly accepted connections. Do you mean a selector is reused > after its sole connection is closed? > > > I'm not sure what you mean by memory chasing. I changed the BitSet to a > HashSet because the file descriptor numbers can get easily get into the > millions. BitSet can allocate a good chunk of memory even if only a couple > of file descriptors are registered with it (if the file descriptor numbers > are large). When you have hundreds of thousands of open Selectors the > memory usage adds up. This is one reason why simply closing Selectors > after a connection is closed does not solve the memory usage problem. > > From my perspective it is the EpollArrayWrapper that is wasting memory > resources. It can allocate memory that is not used and never releases any > that is allocated. Likely this was done for performance reasons. I > haven't looked back through the revision history. > > I'm not proposing the suggested diff to be committed but rather I'm > presenting it to show where the memory is accumulating in the application. > After sun.nio.ch.maxUpdateArraySize was set to a smaller value it was the > 'registered' object and 'eventsHigh' object within EpollArrayWrapper that > were accumulating memory. I'm asking if there is something that can be > done to reduce the memory usage of EpollArrayWrapper by either adding more > system properties to change its behaviour or create a second implementation? > > thanks, > Patrick Bergamin > > > You should bite the bullet and reimplement your application. If I'm > reading right, you're wasting lots of resources all around, and your > proposal is just a hack (with adverse performance due to additional memory > chasing via HashSet) that will likely catch up to you anyway. > > sent from my phone > On Oct 22, 2015 4:51 PM, "Patrick Bergamin" < > patrick.bergamin at wedgenetworks.com> wrote: > >> I'm having problems with memory usage of the current implementation >> of EPollArraySelector in 1.8.0_60 for an existing proxy application. >> We've been on java version 1.7.0_05 for a while now because the new >> implementation of EPollArraySelector does not work well with the >> design of this proxy application. >> >> I did find the sun.nio.ch.maxUpdateArraySize property helped to >> reduce memory usage a bit. But as the proxy application runs >> all the EPollArraySelector objects will slowly accumulate memory. >> >> The current design of the proxy application is to have one Selector >> handle the listen socket. Once a connection is accepted, management >> of the connection is handed off to another thread on another Selector. >> Basically there is one Selector allocated per socket connection. >> The Selectors are never closed they are reused when a new socket >> connection is accepted. >> >> The application was designed before this latest implementation of >> EPollArraySelector. Redesigning the application to decouple the >> Selector from the socket connection would be a fair amount work. >> >> We have machines that are capable of running this proxy application >> with around 350,000 open connections. Since it is a proxy there >> are actually 700,000 open connections. The file descriptor limit >> is set high (5,000,000) to be able to handle all these open socket >> connections. >> >> Below I've included a patch to show what kinds of things I need >> to do to bring the memory usage of EPollArraySelector down for >> this proxy application. >> >> Is there any interest in including a second epoll implementation >> in openjdk that uses less memory or perhaps have more properties >> to control the memory usage of the existing EPollArraySelector? >> >> thanks, >> Patrick Bergamin >> >> --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 >> +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 -0600 >> @@ -29,6 +29,7 @@ >> import java.security.AccessController; >> import java.util.BitSet; >> import java.util.HashMap; >> +import java.util.HashSet; >> import java.util.Map; >> import sun.security.action.GetIntegerAction; >> >> @@ -122,7 +123,7 @@ >> >> // Used by release and updateRegistrations to track whether a file >> // descriptor is registered with epoll. >> - private final BitSet registered = new BitSet(); >> + private final HashSet registered = new HashSet(); >> >> >> EPollArrayWrapper() throws IOException { >> @@ -187,7 +188,10 @@ >> } >> } else { >> Integer key = Integer.valueOf(fd); >> - if (!isEventsHighKilled(key) || force) { >> + if (events == KILLED) { >> + eventsHigh.remove(key); >> + } >> + else if (!isEventsHighKilled(key) || force) { >> eventsHigh.put(key, Byte.valueOf(events)); >> } >> } >> @@ -201,6 +205,9 @@ >> return eventsLow[fd]; >> } else { >> Byte result = eventsHigh.get(Integer.valueOf(fd)); >> + if (result == null) { >> + return KILLED; >> + } >> // result should never be null >> return result.byteValue(); >> } >> @@ -235,7 +242,7 @@ >> // force the initial update events to 0 as it may be KILLED by a >> // previous registration. >> synchronized (updateLock) { >> - assert !registered.get(fd); >> + assert !registered.contains(fd); >> setUpdateEvents(fd, (byte)0, true); >> } >> } >> @@ -249,9 +256,9 @@ >> setUpdateEvents(fd, KILLED, false); >> >> // remove from epoll >> - if (registered.get(fd)) { >> + if (registered.contains(fd)) { >> epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); >> - registered.clear(fd); >> + registered.remove(fd); >> } >> } >> } >> @@ -286,7 +293,7 @@ >> while (j < updateCount) { >> int fd = updateDescriptors[j]; >> short events = getUpdateEvents(fd); >> - boolean isRegistered = registered.get(fd); >> + boolean isRegistered = registered.contains(fd); >> int opcode = 0; >> >> if (events != KILLED) { >> @@ -298,9 +305,9 @@ >> if (opcode != 0) { >> epollCtl(epfd, opcode, fd, events); >> if (opcode == EPOLL_CTL_ADD) { >> - registered.set(fd); >> + registered.add(fd); >> } else if (opcode == EPOLL_CTL_DEL) { >> - registered.clear(fd); >> + registered.remove(fd); >> } >> } >> } >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.bergamin at wedgenetworks.com Fri Oct 23 16:35:49 2015 From: patrick.bergamin at wedgenetworks.com (Patrick Bergamin) Date: Fri, 23 Oct 2015 10:35:49 -0600 Subject: Memory usage of EPollArraySelector In-Reply-To: References: <56294C20.4030300@wedgenetworks.com> <562A5B0B.4080708@wedgenetworks.com> Message-ID: <562A61E5.7020001@wedgenetworks.com> I thought about using one Selector per thread. I wasn't sure if that was going reduce memory usage enough though as the application can allow upwards of 80000 worker threads. I should try this out as it isn't a large change to the application. thanks. On 15-10-23 10:23 AM, Vitaly Davidovich wrote: > The entire problem is you have a Selector per connection rather than > Selector per worker thread, as I mentioned in my previous reply. I > don't think Selector was designed for such a thing, and so using a > BitSet makes sense for the intended usage. HashSet will reduce your > memory footprint because you'll have just 1 entry in there, but the > common/intended case will make it worse as it has terrible memory > locality properties. Your real fix is to redesign the Selector usage > within your application. > > On Fri, Oct 23, 2015 at 12:06 PM, Patrick Bergamin > > wrote: > > On 15-10-22 03:39 PM, Vitaly Davidovich wrote: >> >> Patrick, >> > > Yes Selectors are reused. One Selector is used per connection. > When a socket connection is closed the Selectors are cached along > with a connection handler object so they can be reused when new > socket connections are accepted. > >> I'm confused - how many selectors do you have? You mentioned a >> selector is created for each accepted connection but then state >> that selectors are reused for newly accepted connections. Do you >> mean a selector is reused after its sole connection is closed? >> > > I'm not sure what you mean by memory chasing. I changed the > BitSet to a HashSet because the file descriptor numbers can get > easily get into the millions. BitSet can allocate a good chunk of > memory even if only a couple of file descriptors are registered > with it (if the file descriptor numbers are large). When you have > hundreds of thousands of open Selectors the memory usage adds > up. This is one reason why simply closing Selectors after a > connection is closed does not solve the memory usage problem. > > From my perspective it is the EpollArrayWrapper that is wasting > memory resources. It can allocate memory that is not used and > never releases any that is allocated. Likely this was done for > performance reasons. I haven't looked back through the revision > history. > > I'm not proposing the suggested diff to be committed but rather > I'm presenting it to show where the memory is accumulating in the > application. After sun.nio.ch.maxUpdateArraySize was set to a > smaller value it was the 'registered' object and 'eventsHigh' > object within EpollArrayWrapper that were accumulating memory. I'm > asking if there is something that can be done to reduce the memory > usage of EpollArrayWrapper by either adding more system properties > to change its behaviour or create a second implementation? > > thanks, > Patrick Bergamin > > >> You should bite the bullet and reimplement your application. If >> I'm reading right, you're wasting lots of resources all around, >> and your proposal is just a hack (with adverse performance due to >> additional memory chasing via HashSet) that will likely catch up >> to you anyway. >> >> sent from my phone >> >> On Oct 22, 2015 4:51 PM, "Patrick Bergamin" >> > > wrote: >> >> I'm having problems with memory usage of the current >> implementation >> of EPollArraySelector in 1.8.0_60 for an existing proxy >> application. >> We've been on java version 1.7.0_05 for a while now because >> the new >> implementation of EPollArraySelector does not work well with the >> design of this proxy application. >> >> I did find the sun.nio.ch.maxUpdateArraySize property helped to >> reduce memory usage a bit. But as the proxy application runs >> all the EPollArraySelector objects will slowly accumulate memory. >> >> The current design of the proxy application is to have one >> Selector >> handle the listen socket. Once a connection is accepted, >> management >> of the connection is handed off to another thread on another >> Selector. >> Basically there is one Selector allocated per socket connection. >> The Selectors are never closed they are reused when a new socket >> connection is accepted. >> >> The application was designed before this latest implementation of >> EPollArraySelector. Redesigning the application to decouple the >> Selector from the socket connection would be a fair amount work. >> >> We have machines that are capable of running this proxy >> application >> with around 350,000 open connections. Since it is a proxy there >> are actually 700,000 open connections. The file descriptor limit >> is set high (5,000,000) to be able to handle all these open >> socket >> connections. >> >> Below I've included a patch to show what kinds of things I need >> to do to bring the memory usage of EPollArraySelector down for >> this proxy application. >> >> Is there any interest in including a second epoll implementation >> in openjdk that uses less memory or perhaps have more properties >> to control the memory usage of the existing EPollArraySelector? >> >> thanks, >> Patrick Bergamin >> >> --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 >> +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 >> -0600 >> @@ -29,6 +29,7 @@ >> import java.security.AccessController; >> import java.util.BitSet; >> import java.util.HashMap; >> +import java.util.HashSet; >> import java.util.Map; >> import sun.security.action.GetIntegerAction; >> >> @@ -122,7 +123,7 @@ >> >> // Used by release and updateRegistrations to track >> whether a file >> // descriptor is registered with epoll. >> - private final BitSet registered = new BitSet(); >> + private final HashSet registered = new >> HashSet(); >> >> >> EPollArrayWrapper() throws IOException { >> @@ -187,7 +188,10 @@ >> } >> } else { >> Integer key = Integer.valueOf(fd); >> - if (!isEventsHighKilled(key) || force) { >> + if (events == KILLED) { >> + eventsHigh.remove(key); >> + } >> + else if (!isEventsHighKilled(key) || force) { >> eventsHigh.put(key, Byte.valueOf(events)); >> } >> } >> @@ -201,6 +205,9 @@ >> return eventsLow[fd]; >> } else { >> Byte result = eventsHigh.get(Integer.valueOf(fd)); >> + if (result == null) { >> + return KILLED; >> + } >> // result should never be null >> return result.byteValue(); >> } >> @@ -235,7 +242,7 @@ >> // force the initial update events to 0 as it may be >> KILLED by a >> // previous registration. >> synchronized (updateLock) { >> - assert !registered.get(fd); >> + assert !registered.contains(fd); >> setUpdateEvents(fd, (byte)0, true); >> } >> } >> @@ -249,9 +256,9 @@ >> setUpdateEvents(fd, KILLED, false); >> >> // remove from epoll >> - if (registered.get(fd)) { >> + if (registered.contains(fd)) { >> epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); >> - registered.clear(fd); >> + registered.remove(fd); >> } >> } >> } >> @@ -286,7 +293,7 @@ >> while (j < updateCount) { >> int fd = updateDescriptors[j]; >> short events = getUpdateEvents(fd); >> - boolean isRegistered = registered.get(fd); >> + boolean isRegistered = registered.contains(fd); >> int opcode = 0; >> >> if (events != KILLED) { >> @@ -298,9 +305,9 @@ >> if (opcode != 0) { >> epollCtl(epfd, opcode, fd, events); >> if (opcode == EPOLL_CTL_ADD) { >> - registered.set(fd); >> + registered.add(fd); >> } else if (opcode == EPOLL_CTL_DEL) { >> - registered.clear(fd); >> + registered.remove(fd); >> } >> } >> } >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitalyd at gmail.com Fri Oct 23 16:45:10 2015 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Fri, 23 Oct 2015 12:45:10 -0400 Subject: Memory usage of EPollArraySelector In-Reply-To: <562A61E5.7020001@wedgenetworks.com> References: <56294C20.4030300@wedgenetworks.com> <562A5B0B.4080708@wedgenetworks.com> <562A61E5.7020001@wedgenetworks.com> Message-ID: 80k worker threads?? That doesn't sound right either :). Roughly speaking, I suggest the following: 1) N compute bound worker threads where N = # of cpus 2) 1-2 i/o threads that monitor fd's for write/read readiness, and perform the read/write operations (workers in #1 hand off data to these) 3) Some threadpool for IO/blocking operations where you don't have async options (e.g. filesystem/disk) which you can size depending on latency of the i/o operations Do some research on modern i/o threading models (e.g. nginx, netty, etc). It may be a larger effort, but you'll be happier in the long run. On Fri, Oct 23, 2015 at 12:35 PM, Patrick Bergamin < patrick.bergamin at wedgenetworks.com> wrote: > I thought about using one Selector per thread. I wasn't sure if that was > going reduce memory usage enough though as the application can allow > upwards of 80000 worker threads. I should try this out as it isn't a large > change to the application. > > thanks. > > > > On 15-10-23 10:23 AM, Vitaly Davidovich wrote: > > The entire problem is you have a Selector per connection rather than > Selector per worker thread, as I mentioned in my previous reply. I don't > think Selector was designed for such a thing, and so using a BitSet makes > sense for the intended usage. HashSet will reduce your memory footprint > because you'll have just 1 entry in there, but the common/intended case > will make it worse as it has terrible memory locality properties. Your > real fix is to redesign the Selector usage within your application. > > On Fri, Oct 23, 2015 at 12:06 PM, Patrick Bergamin < > patrick.bergamin at wedgenetworks.com> > wrote: > >> On 15-10-22 03:39 PM, Vitaly Davidovich wrote: >> >> Patrick, >> >> >> Yes Selectors are reused. One Selector is used per connection. When a >> socket connection is closed the Selectors are cached along with a >> connection handler object so they can be reused when new socket connections >> are accepted. >> >> I'm confused - how many selectors do you have? You mentioned a selector >> is created for each accepted connection but then state that selectors are >> reused for newly accepted connections. Do you mean a selector is reused >> after its sole connection is closed? >> >> >> I'm not sure what you mean by memory chasing. I changed the BitSet to a >> HashSet because the file descriptor numbers can get easily get into the >> millions. BitSet can allocate a good chunk of memory even if only a couple >> of file descriptors are registered with it (if the file descriptor numbers >> are large). When you have hundreds of thousands of open Selectors the >> memory usage adds up. This is one reason why simply closing Selectors >> after a connection is closed does not solve the memory usage problem. >> >> From my perspective it is the EpollArrayWrapper that is wasting memory >> resources. It can allocate memory that is not used and never releases any >> that is allocated. Likely this was done for performance reasons. I >> haven't looked back through the revision history. >> >> I'm not proposing the suggested diff to be committed but rather I'm >> presenting it to show where the memory is accumulating in the application. >> After sun.nio.ch.maxUpdateArraySize was set to a smaller value it was the >> 'registered' object and 'eventsHigh' object within EpollArrayWrapper that >> were accumulating memory. I'm asking if there is something that can be >> done to reduce the memory usage of EpollArrayWrapper by either adding more >> system properties to change its behaviour or create a second implementation? >> >> thanks, >> Patrick Bergamin >> >> >> You should bite the bullet and reimplement your application. If I'm >> reading right, you're wasting lots of resources all around, and your >> proposal is just a hack (with adverse performance due to additional memory >> chasing via HashSet) that will likely catch up to you anyway. >> >> sent from my phone >> On Oct 22, 2015 4:51 PM, "Patrick Bergamin" < >> patrick.bergamin at wedgenetworks.com> wrote: >> >>> I'm having problems with memory usage of the current implementation >>> of EPollArraySelector in 1.8.0_60 for an existing proxy application. >>> We've been on java version 1.7.0_05 for a while now because the new >>> implementation of EPollArraySelector does not work well with the >>> design of this proxy application. >>> >>> I did find the sun.nio.ch.maxUpdateArraySize property helped to >>> reduce memory usage a bit. But as the proxy application runs >>> all the EPollArraySelector objects will slowly accumulate memory. >>> >>> The current design of the proxy application is to have one Selector >>> handle the listen socket. Once a connection is accepted, management >>> of the connection is handed off to another thread on another Selector. >>> Basically there is one Selector allocated per socket connection. >>> The Selectors are never closed they are reused when a new socket >>> connection is accepted. >>> >>> The application was designed before this latest implementation of >>> EPollArraySelector. Redesigning the application to decouple the >>> Selector from the socket connection would be a fair amount work. >>> >>> We have machines that are capable of running this proxy application >>> with around 350,000 open connections. Since it is a proxy there >>> are actually 700,000 open connections. The file descriptor limit >>> is set high (5,000,000) to be able to handle all these open socket >>> connections. >>> >>> Below I've included a patch to show what kinds of things I need >>> to do to bring the memory usage of EPollArraySelector down for >>> this proxy application. >>> >>> Is there any interest in including a second epoll implementation >>> in openjdk that uses less memory or perhaps have more properties >>> to control the memory usage of the existing EPollArraySelector? >>> >>> thanks, >>> Patrick Bergamin >>> >>> --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 >>> +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 -0600 >>> @@ -29,6 +29,7 @@ >>> import java.security.AccessController; >>> import java.util.BitSet; >>> import java.util.HashMap; >>> +import java.util.HashSet; >>> import java.util.Map; >>> import sun.security.action.GetIntegerAction; >>> >>> @@ -122,7 +123,7 @@ >>> >>> // Used by release and updateRegistrations to track whether a file >>> // descriptor is registered with epoll. >>> - private final BitSet registered = new BitSet(); >>> + private final HashSet registered = new HashSet(); >>> >>> >>> EPollArrayWrapper() throws IOException { >>> @@ -187,7 +188,10 @@ >>> } >>> } else { >>> Integer key = Integer.valueOf(fd); >>> - if (!isEventsHighKilled(key) || force) { >>> + if (events == KILLED) { >>> + eventsHigh.remove(key); >>> + } >>> + else if (!isEventsHighKilled(key) || force) { >>> eventsHigh.put(key, Byte.valueOf(events)); >>> } >>> } >>> @@ -201,6 +205,9 @@ >>> return eventsLow[fd]; >>> } else { >>> Byte result = eventsHigh.get(Integer.valueOf(fd)); >>> + if (result == null) { >>> + return KILLED; >>> + } >>> // result should never be null >>> return result.byteValue(); >>> } >>> @@ -235,7 +242,7 @@ >>> // force the initial update events to 0 as it may be KILLED by a >>> // previous registration. >>> synchronized (updateLock) { >>> - assert !registered.get(fd); >>> + assert !registered.contains(fd); >>> setUpdateEvents(fd, (byte)0, true); >>> } >>> } >>> @@ -249,9 +256,9 @@ >>> setUpdateEvents(fd, KILLED, false); >>> >>> // remove from epoll >>> - if (registered.get(fd)) { >>> + if (registered.contains(fd)) { >>> epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); >>> - registered.clear(fd); >>> + registered.remove(fd); >>> } >>> } >>> } >>> @@ -286,7 +293,7 @@ >>> while (j < updateCount) { >>> int fd = updateDescriptors[j]; >>> short events = getUpdateEvents(fd); >>> - boolean isRegistered = registered.get(fd); >>> + boolean isRegistered = registered.contains(fd); >>> int opcode = 0; >>> >>> if (events != KILLED) { >>> @@ -298,9 +305,9 @@ >>> if (opcode != 0) { >>> epollCtl(epfd, opcode, fd, events); >>> if (opcode == EPOLL_CTL_ADD) { >>> - registered.set(fd); >>> + registered.add(fd); >>> } else if (opcode == EPOLL_CTL_DEL) { >>> - registered.clear(fd); >>> + registered.remove(fd); >>> } >>> } >>> } >>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrick.bergamin at wedgenetworks.com Fri Oct 23 17:23:41 2015 From: patrick.bergamin at wedgenetworks.com (Patrick Bergamin) Date: Fri, 23 Oct 2015 11:23:41 -0600 Subject: Memory usage of EPollArraySelector In-Reply-To: References: <56294C20.4030300@wedgenetworks.com> <562A5B0B.4080708@wedgenetworks.com> <562A61E5.7020001@wedgenetworks.com> Message-ID: <562A6D1D.5060307@wedgenetworks.com> Hi Vitaly, Yeah I understand what your saying. Right now we want to move forward with a newer version of java. This application is holding us back. Decoupling the Selectors from the socket connections is not a small task. All your suggestions are good ideas and match some of the ideas I have had on reworking the application. thanks for the help. On 15-10-23 10:45 AM, Vitaly Davidovich wrote: > 80k worker threads?? That doesn't sound right either :). Roughly > speaking, I suggest the following: > > 1) N compute bound worker threads where N = # of cpus > 2) 1-2 i/o threads that monitor fd's for write/read readiness, and > perform the read/write operations (workers in #1 hand off data to these) > 3) Some threadpool for IO/blocking operations where you don't have > async options (e.g. filesystem/disk) which you can size depending on > latency of the i/o operations > > Do some research on modern i/o threading models (e.g. nginx, netty, > etc). It may be a larger effort, but you'll be happier in the long run. > > > > On Fri, Oct 23, 2015 at 12:35 PM, Patrick Bergamin > > wrote: > > I thought about using one Selector per thread. I wasn't sure if > that was going reduce memory usage enough though as the > application can allow upwards of 80000 worker threads. I should > try this out as it isn't a large change to the application. > > thanks. > > > > On 15-10-23 10:23 AM, Vitaly Davidovich wrote: >> The entire problem is you have a Selector per connection rather >> than Selector per worker thread, as I mentioned in my previous >> reply. I don't think Selector was designed for such a thing, and >> so using a BitSet makes sense for the intended usage. HashSet >> will reduce your memory footprint because you'll have just 1 >> entry in there, but the common/intended case will make it worse >> as it has terrible memory locality properties. Your real fix is >> to redesign the Selector usage within your application. >> >> On Fri, Oct 23, 2015 at 12:06 PM, Patrick Bergamin >> > > wrote: >> >> On 15-10-22 03:39 PM, Vitaly Davidovich wrote: >>> >>> Patrick, >>> >> >> Yes Selectors are reused. One Selector is used per >> connection. When a socket connection is closed the Selectors >> are cached along with a connection handler object so they can >> be reused when new socket connections are accepted. >> >>> I'm confused - how many selectors do you have? You mentioned >>> a selector is created for each accepted connection but then >>> state that selectors are reused for newly accepted >>> connections. Do you mean a selector is reused after its >>> sole connection is closed? >>> >> >> I'm not sure what you mean by memory chasing. I changed the >> BitSet to a HashSet because the file descriptor numbers can >> get easily get into the millions. BitSet can allocate a good >> chunk of memory even if only a couple of file descriptors are >> registered with it (if the file descriptor numbers are >> large). When you have hundreds of thousands of open >> Selectors the memory usage adds up. This is one reason why >> simply closing Selectors after a connection is closed does >> not solve the memory usage problem. >> >> From my perspective it is the EpollArrayWrapper that is >> wasting memory resources. It can allocate memory that is >> not used and never releases any that is allocated. Likely >> this was done for performance reasons. I haven't looked back >> through the revision history. >> >> I'm not proposing the suggested diff to be committed but >> rather I'm presenting it to show where the memory is >> accumulating in the application. After >> sun.nio.ch.maxUpdateArraySize was set to a smaller value it >> was the 'registered' object and 'eventsHigh' object within >> EpollArrayWrapper that were accumulating memory. I'm asking >> if there is something that can be done to reduce the memory >> usage of EpollArrayWrapper by either adding more system >> properties to change its behaviour or create a second >> implementation? >> >> thanks, >> Patrick Bergamin >> >> >>> You should bite the bullet and reimplement your >>> application. If I'm reading right, you're wasting lots of >>> resources all around, and your proposal is just a hack (with >>> adverse performance due to additional memory chasing via >>> HashSet) that will likely catch up to you anyway. >>> >>> sent from my phone >>> >>> On Oct 22, 2015 4:51 PM, "Patrick Bergamin" >>> >> > wrote: >>> >>> I'm having problems with memory usage of the current >>> implementation >>> of EPollArraySelector in 1.8.0_60 for an existing proxy >>> application. >>> We've been on java version 1.7.0_05 for a while now >>> because the new >>> implementation of EPollArraySelector does not work well >>> with the >>> design of this proxy application. >>> >>> I did find the sun.nio.ch.maxUpdateArraySize property >>> helped to >>> reduce memory usage a bit. But as the proxy application >>> runs >>> all the EPollArraySelector objects will slowly >>> accumulate memory. >>> >>> The current design of the proxy application is to have >>> one Selector >>> handle the listen socket. Once a connection is >>> accepted, management >>> of the connection is handed off to another thread on >>> another Selector. >>> Basically there is one Selector allocated per socket >>> connection. >>> The Selectors are never closed they are reused when a >>> new socket >>> connection is accepted. >>> >>> The application was designed before this latest >>> implementation of >>> EPollArraySelector. Redesigning the application to >>> decouple the >>> Selector from the socket connection would be a fair >>> amount work. >>> >>> We have machines that are capable of running this proxy >>> application >>> with around 350,000 open connections. Since it is a >>> proxy there >>> are actually 700,000 open connections. The file >>> descriptor limit >>> is set high (5,000,000) to be able to handle all these >>> open socket >>> connections. >>> >>> Below I've included a patch to show what kinds of things >>> I need >>> to do to bring the memory usage of EPollArraySelector >>> down for >>> this proxy application. >>> >>> Is there any interest in including a second epoll >>> implementation >>> in openjdk that uses less memory or perhaps have more >>> properties >>> to control the memory usage of the existing >>> EPollArraySelector? >>> >>> thanks, >>> Patrick Bergamin >>> >>> --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 >>> -0600 >>> +++ EPollArrayWrapper2.java 2015-09-28 >>> 15:31:41.712607415 -0600 >>> @@ -29,6 +29,7 @@ >>> import java.security.AccessController; >>> import java.util.BitSet; >>> import java.util.HashMap; >>> +import java.util.HashSet; >>> import java.util.Map; >>> import sun.security.action.GetIntegerAction; >>> >>> @@ -122,7 +123,7 @@ >>> >>> // Used by release and updateRegistrations to track >>> whether a file >>> // descriptor is registered with epoll. >>> - private final BitSet registered = new BitSet(); >>> + private final HashSet registered = new >>> HashSet(); >>> >>> >>> EPollArrayWrapper() throws IOException { >>> @@ -187,7 +188,10 @@ >>> } >>> } else { >>> Integer key = Integer.valueOf(fd); >>> - if (!isEventsHighKilled(key) || force) { >>> + if (events == KILLED) { >>> + eventsHigh.remove(key); >>> + } >>> + else if (!isEventsHighKilled(key) || force) { >>> eventsHigh.put(key, Byte.valueOf(events)); >>> } >>> } >>> @@ -201,6 +205,9 @@ >>> return eventsLow[fd]; >>> } else { >>> Byte result = >>> eventsHigh.get(Integer.valueOf(fd)); >>> + if (result == null) { >>> + return KILLED; >>> + } >>> // result should never be null >>> return result.byteValue(); >>> } >>> @@ -235,7 +242,7 @@ >>> // force the initial update events to 0 as it >>> may be KILLED by a >>> // previous registration. >>> synchronized (updateLock) { >>> - assert !registered.get(fd); >>> + assert !registered.contains(fd); >>> setUpdateEvents(fd, (byte)0, true); >>> } >>> } >>> @@ -249,9 +256,9 @@ >>> setUpdateEvents(fd, KILLED, false); >>> >>> // remove from epoll >>> - if (registered.get(fd)) { >>> + if (registered.contains(fd)) { >>> epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); >>> - registered.clear(fd); >>> + registered.remove(fd); >>> } >>> } >>> } >>> @@ -286,7 +293,7 @@ >>> while (j < updateCount) { >>> int fd = updateDescriptors[j]; >>> short events = getUpdateEvents(fd); >>> - boolean isRegistered = registered.get(fd); >>> + boolean isRegistered = >>> registered.contains(fd); >>> int opcode = 0; >>> >>> if (events != KILLED) { >>> @@ -298,9 +305,9 @@ >>> if (opcode != 0) { >>> epollCtl(epfd, opcode, fd, events); >>> if (opcode == EPOLL_CTL_ADD) { >>> - registered.set(fd); >>> + registered.add(fd); >>> } else if (opcode == >>> EPOLL_CTL_DEL) { >>> - registered.clear(fd); >>> + registered.remove(fd); >>> } >>> } >>> } >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitalyd at gmail.com Fri Oct 23 17:29:22 2015 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Fri, 23 Oct 2015 13:29:22 -0400 Subject: Memory usage of EPollArraySelector In-Reply-To: <562A6D1D.5060307@wedgenetworks.com> References: <56294C20.4030300@wedgenetworks.com> <562A5B0B.4080708@wedgenetworks.com> <562A61E5.7020001@wedgenetworks.com> <562A6D1D.5060307@wedgenetworks.com> Message-ID: Yes, I apologize for not really answering your question, but I suspect you will not get much traction since the proposed changes cater to poor initial design (and unintended use of selectors). Perhaps you can look into writing a custom selector provider that would return your modified epoll one. On Fri, Oct 23, 2015 at 1:23 PM, Patrick Bergamin < patrick.bergamin at wedgenetworks.com> wrote: > Hi Vitaly, > > Yeah I understand what your saying. Right now we want to move forward > with a newer version of java. This application is holding us back. > Decoupling the Selectors from the socket connections is not a small task. > All your suggestions are good ideas and match some of the ideas I have had > on reworking the application. > > thanks for the help. > > > On 15-10-23 10:45 AM, Vitaly Davidovich wrote: > > 80k worker threads?? That doesn't sound right either :). Roughly > speaking, I suggest the following: > > 1) N compute bound worker threads where N = # of cpus > 2) 1-2 i/o threads that monitor fd's for write/read readiness, and perform > the read/write operations (workers in #1 hand off data to these) > 3) Some threadpool for IO/blocking operations where you don't have async > options (e.g. filesystem/disk) which you can size depending on latency of > the i/o operations > > Do some research on modern i/o threading models (e.g. nginx, netty, etc). > It may be a larger effort, but you'll be happier in the long run. > > > > On Fri, Oct 23, 2015 at 12:35 PM, Patrick Bergamin < > patrick.bergamin at wedgenetworks.com> > wrote: > >> I thought about using one Selector per thread. I wasn't sure if that was >> going reduce memory usage enough though as the application can allow >> upwards of 80000 worker threads. I should try this out as it isn't a large >> change to the application. >> >> thanks. >> >> >> >> On 15-10-23 10:23 AM, Vitaly Davidovich wrote: >> >> The entire problem is you have a Selector per connection rather than >> Selector per worker thread, as I mentioned in my previous reply. I don't >> think Selector was designed for such a thing, and so using a BitSet makes >> sense for the intended usage. HashSet will reduce your memory footprint >> because you'll have just 1 entry in there, but the common/intended case >> will make it worse as it has terrible memory locality properties. Your >> real fix is to redesign the Selector usage within your application. >> >> On Fri, Oct 23, 2015 at 12:06 PM, Patrick Bergamin < >> patrick.bergamin at wedgenetworks.com> >> wrote: >> >>> On 15-10-22 03:39 PM, Vitaly Davidovich wrote: >>> >>> Patrick, >>> >>> >>> Yes Selectors are reused. One Selector is used per connection. When a >>> socket connection is closed the Selectors are cached along with a >>> connection handler object so they can be reused when new socket connections >>> are accepted. >>> >>> I'm confused - how many selectors do you have? You mentioned a selector >>> is created for each accepted connection but then state that selectors are >>> reused for newly accepted connections. Do you mean a selector is reused >>> after its sole connection is closed? >>> >>> >>> I'm not sure what you mean by memory chasing. I changed the BitSet to a >>> HashSet because the file descriptor numbers can get easily get into the >>> millions. BitSet can allocate a good chunk of memory even if only a couple >>> of file descriptors are registered with it (if the file descriptor numbers >>> are large). When you have hundreds of thousands of open Selectors the >>> memory usage adds up. This is one reason why simply closing Selectors >>> after a connection is closed does not solve the memory usage problem. >>> >>> From my perspective it is the EpollArrayWrapper that is wasting memory >>> resources. It can allocate memory that is not used and never releases any >>> that is allocated. Likely this was done for performance reasons. I >>> haven't looked back through the revision history. >>> >>> I'm not proposing the suggested diff to be committed but rather I'm >>> presenting it to show where the memory is accumulating in the application. >>> After sun.nio.ch.maxUpdateArraySize was set to a smaller value it was the >>> 'registered' object and 'eventsHigh' object within EpollArrayWrapper that >>> were accumulating memory. I'm asking if there is something that can be >>> done to reduce the memory usage of EpollArrayWrapper by either adding more >>> system properties to change its behaviour or create a second implementation? >>> >>> thanks, >>> Patrick Bergamin >>> >>> >>> You should bite the bullet and reimplement your application. If I'm >>> reading right, you're wasting lots of resources all around, and your >>> proposal is just a hack (with adverse performance due to additional memory >>> chasing via HashSet) that will likely catch up to you anyway. >>> >>> sent from my phone >>> On Oct 22, 2015 4:51 PM, "Patrick Bergamin" < >>> patrick.bergamin at wedgenetworks.com> wrote: >>> >>>> I'm having problems with memory usage of the current implementation >>>> of EPollArraySelector in 1.8.0_60 for an existing proxy application. >>>> We've been on java version 1.7.0_05 for a while now because the new >>>> implementation of EPollArraySelector does not work well with the >>>> design of this proxy application. >>>> >>>> I did find the sun.nio.ch.maxUpdateArraySize property helped to >>>> reduce memory usage a bit. But as the proxy application runs >>>> all the EPollArraySelector objects will slowly accumulate memory. >>>> >>>> The current design of the proxy application is to have one Selector >>>> handle the listen socket. Once a connection is accepted, management >>>> of the connection is handed off to another thread on another Selector. >>>> Basically there is one Selector allocated per socket connection. >>>> The Selectors are never closed they are reused when a new socket >>>> connection is accepted. >>>> >>>> The application was designed before this latest implementation of >>>> EPollArraySelector. Redesigning the application to decouple the >>>> Selector from the socket connection would be a fair amount work. >>>> >>>> We have machines that are capable of running this proxy application >>>> with around 350,000 open connections. Since it is a proxy there >>>> are actually 700,000 open connections. The file descriptor limit >>>> is set high (5,000,000) to be able to handle all these open socket >>>> connections. >>>> >>>> Below I've included a patch to show what kinds of things I need >>>> to do to bring the memory usage of EPollArraySelector down for >>>> this proxy application. >>>> >>>> Is there any interest in including a second epoll implementation >>>> in openjdk that uses less memory or perhaps have more properties >>>> to control the memory usage of the existing EPollArraySelector? >>>> >>>> thanks, >>>> Patrick Bergamin >>>> >>>> --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 >>>> +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 -0600 >>>> @@ -29,6 +29,7 @@ >>>> import java.security.AccessController; >>>> import java.util.BitSet; >>>> import java.util.HashMap; >>>> +import java.util.HashSet; >>>> import java.util.Map; >>>> import sun.security.action.GetIntegerAction; >>>> >>>> @@ -122,7 +123,7 @@ >>>> >>>> // Used by release and updateRegistrations to track whether a file >>>> // descriptor is registered with epoll. >>>> - private final BitSet registered = new BitSet(); >>>> + private final HashSet registered = new HashSet(); >>>> >>>> >>>> EPollArrayWrapper() throws IOException { >>>> @@ -187,7 +188,10 @@ >>>> } >>>> } else { >>>> Integer key = Integer.valueOf(fd); >>>> - if (!isEventsHighKilled(key) || force) { >>>> + if (events == KILLED) { >>>> + eventsHigh.remove(key); >>>> + } >>>> + else if (!isEventsHighKilled(key) || force) { >>>> eventsHigh.put(key, Byte.valueOf(events)); >>>> } >>>> } >>>> @@ -201,6 +205,9 @@ >>>> return eventsLow[fd]; >>>> } else { >>>> Byte result = eventsHigh.get(Integer.valueOf(fd)); >>>> + if (result == null) { >>>> + return KILLED; >>>> + } >>>> // result should never be null >>>> return result.byteValue(); >>>> } >>>> @@ -235,7 +242,7 @@ >>>> // force the initial update events to 0 as it may be KILLED by >>>> a >>>> // previous registration. >>>> synchronized (updateLock) { >>>> - assert !registered.get(fd); >>>> + assert !registered.contains(fd); >>>> setUpdateEvents(fd, (byte)0, true); >>>> } >>>> } >>>> @@ -249,9 +256,9 @@ >>>> setUpdateEvents(fd, KILLED, false); >>>> >>>> // remove from epoll >>>> - if (registered.get(fd)) { >>>> + if (registered.contains(fd)) { >>>> epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); >>>> - registered.clear(fd); >>>> + registered.remove(fd); >>>> } >>>> } >>>> } >>>> @@ -286,7 +293,7 @@ >>>> while (j < updateCount) { >>>> int fd = updateDescriptors[j]; >>>> short events = getUpdateEvents(fd); >>>> - boolean isRegistered = registered.get(fd); >>>> + boolean isRegistered = registered.contains(fd); >>>> int opcode = 0; >>>> >>>> if (events != KILLED) { >>>> @@ -298,9 +305,9 @@ >>>> if (opcode != 0) { >>>> epollCtl(epfd, opcode, fd, events); >>>> if (opcode == EPOLL_CTL_ADD) { >>>> - registered.set(fd); >>>> + registered.add(fd); >>>> } else if (opcode == EPOLL_CTL_DEL) { >>>> - registered.clear(fd); >>>> + registered.remove(fd); >>>> } >>>> } >>>> } >>>> >>>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From weijun.wang at oracle.com Sat Oct 24 14:26:30 2015 From: weijun.wang at oracle.com (Wang Weijun) Date: Sat, 24 Oct 2015 22:26:30 +0800 Subject: Strange "..\" on Windows In-Reply-To: <562A000E.1010908@oracle.com> References: <562A000E.1010908@oracle.com> Message-ID: <59BEB424-9ECE-45E3-ADF5-B62F81B6E3A7@oracle.com> I found 2 problems in relativize() of WindowsPath.java: 401 // skip matching names .... now "i" points to the first non match. 410 // append ..\ for remaining names in the base 411 StringBuilder result = new StringBuilder(); 412 for (int j=i; j On Oct 23, 2015, at 5:38 PM, Weijun Wang wrote: > > Paths.get("x").relativize(Paths.get("")) returns ..\ on Windows, which is not the same as Paths.get("..\\") (which shows as ..). In fact, its getName(0).toString() is exactly ..\. > > It looks like the \ is not treated as a path separator. > > Also, Paths.get("x/y").relativize(Paths.get("")) returns ..\..\ on Windows, but its getName(0) is .. and getName(1) is ..\ > > On Mac, it's all normal. > > Is this a bug? > > Thanks > Max > > p.s. I run in cygwin of Windows 10. > > From weijun.wang at oracle.com Sat Oct 24 14:29:25 2015 From: weijun.wang at oracle.com (Wang Weijun) Date: Sat, 24 Oct 2015 22:29:25 +0800 Subject: Strange "..\" on Windows In-Reply-To: <562A000E.1010908@oracle.com> References: <562A000E.1010908@oracle.com> Message-ID: <800270CA-BE05-4B49-BBA3-C188CE0748E6@oracle.com> I found 2 problems in relativize() of WindowsPath.java: 401 // skip matching names .... now "i" points to the first non match. 410 // append ..\ for remaining names in the base 411 StringBuilder result = new StringBuilder(); 412 for (int j=i; j On Oct 23, 2015, at 5:38 PM, Weijun Wang wrote: > > Paths.get("x").relativize(Paths.get("")) returns ..\ on Windows, which is not the same as Paths.get("..\\") (which shows as ..). In fact, its getName(0).toString() is exactly ..\. > > It looks like the \ is not treated as a path separator. > > Also, Paths.get("x/y").relativize(Paths.get("")) returns ..\..\ on Windows, but its getName(0) is .. and getName(1) is ..\ > > On Mac, it's all normal. > > Is this a bug? > > Thanks > Max > > p.s. I run in cygwin of Windows 10. > > From weijun.wang at oracle.com Sat Oct 24 14:51:56 2015 From: weijun.wang at oracle.com (Wang Weijun) Date: Sat, 24 Oct 2015 22:51:56 +0800 Subject: Strange "..\" on Windows In-Reply-To: <800270CA-BE05-4B49-BBA3-C188CE0748E6@oracle.com> References: <562A000E.1010908@oracle.com> <800270CA-BE05-4B49-BBA3-C188CE0748E6@oracle.com> Message-ID: <3FB75674-76D9-4E10-8366-38FBBEFE5FD1@oracle.com> https://bugs.openjdk.java.net/browse/JDK-8140449 filed. --Max > On Oct 24, 2015, at 10:29 PM, Wang Weijun wrote: > > However, if it's only "x".relative(""), it becomes "..\\" + "\\" - "\\", so there is still one trailing slash. From martinrb at google.com Tue Oct 27 20:31:18 2015 From: martinrb at google.com (Martin Buchholz) Date: Tue, 27 Oct 2015 13:31:18 -0700 Subject: Memory usage of EPollArraySelector In-Reply-To: <562A6D1D.5060307@wedgenetworks.com> References: <56294C20.4030300@wedgenetworks.com> <562A5B0B.4080708@wedgenetworks.com> <562A61E5.7020001@wedgenetworks.com> <562A6D1D.5060307@wedgenetworks.com> Message-ID: I agree that the amount of memory retained in the jdk (linear in the file descriptor limit) is a problem with the jdk that should be addressed somehow. Is there an open bug? On Fri, Oct 23, 2015 at 10:23 AM, Patrick Bergamin < patrick.bergamin at wedgenetworks.com> wrote: > Hi Vitaly, > > Yeah I understand what your saying. Right now we want to move forward > with a newer version of java. This application is holding us back. > Decoupling the Selectors from the socket connections is not a small task. > All your suggestions are good ideas and match some of the ideas I have had > on reworking the application. > > thanks for the help. > > > On 15-10-23 10:45 AM, Vitaly Davidovich wrote: > > 80k worker threads?? That doesn't sound right either :). Roughly > speaking, I suggest the following: > > 1) N compute bound worker threads where N = # of cpus > 2) 1-2 i/o threads that monitor fd's for write/read readiness, and perform > the read/write operations (workers in #1 hand off data to these) > 3) Some threadpool for IO/blocking operations where you don't have async > options (e.g. filesystem/disk) which you can size depending on latency of > the i/o operations > > Do some research on modern i/o threading models (e.g. nginx, netty, etc). > It may be a larger effort, but you'll be happier in the long run. > > > > On Fri, Oct 23, 2015 at 12:35 PM, Patrick Bergamin < > patrick.bergamin at wedgenetworks.com> > wrote: > >> I thought about using one Selector per thread. I wasn't sure if that was >> going reduce memory usage enough though as the application can allow >> upwards of 80000 worker threads. I should try this out as it isn't a large >> change to the application. >> >> thanks. >> >> >> >> On 15-10-23 10:23 AM, Vitaly Davidovich wrote: >> >> The entire problem is you have a Selector per connection rather than >> Selector per worker thread, as I mentioned in my previous reply. I don't >> think Selector was designed for such a thing, and so using a BitSet makes >> sense for the intended usage. HashSet will reduce your memory footprint >> because you'll have just 1 entry in there, but the common/intended case >> will make it worse as it has terrible memory locality properties. Your >> real fix is to redesign the Selector usage within your application. >> >> On Fri, Oct 23, 2015 at 12:06 PM, Patrick Bergamin < >> patrick.bergamin at wedgenetworks.com> >> wrote: >> >>> On 15-10-22 03:39 PM, Vitaly Davidovich wrote: >>> >>> Patrick, >>> >>> >>> Yes Selectors are reused. One Selector is used per connection. When a >>> socket connection is closed the Selectors are cached along with a >>> connection handler object so they can be reused when new socket connections >>> are accepted. >>> >>> I'm confused - how many selectors do you have? You mentioned a selector >>> is created for each accepted connection but then state that selectors are >>> reused for newly accepted connections. Do you mean a selector is reused >>> after its sole connection is closed? >>> >>> >>> I'm not sure what you mean by memory chasing. I changed the BitSet to a >>> HashSet because the file descriptor numbers can get easily get into the >>> millions. BitSet can allocate a good chunk of memory even if only a couple >>> of file descriptors are registered with it (if the file descriptor numbers >>> are large). When you have hundreds of thousands of open Selectors the >>> memory usage adds up. This is one reason why simply closing Selectors >>> after a connection is closed does not solve the memory usage problem. >>> >>> From my perspective it is the EpollArrayWrapper that is wasting memory >>> resources. It can allocate memory that is not used and never releases any >>> that is allocated. Likely this was done for performance reasons. I >>> haven't looked back through the revision history. >>> >>> I'm not proposing the suggested diff to be committed but rather I'm >>> presenting it to show where the memory is accumulating in the application. >>> After sun.nio.ch.maxUpdateArraySize was set to a smaller value it was the >>> 'registered' object and 'eventsHigh' object within EpollArrayWrapper that >>> were accumulating memory. I'm asking if there is something that can be >>> done to reduce the memory usage of EpollArrayWrapper by either adding more >>> system properties to change its behaviour or create a second implementation? >>> >>> thanks, >>> Patrick Bergamin >>> >>> >>> You should bite the bullet and reimplement your application. If I'm >>> reading right, you're wasting lots of resources all around, and your >>> proposal is just a hack (with adverse performance due to additional memory >>> chasing via HashSet) that will likely catch up to you anyway. >>> >>> sent from my phone >>> On Oct 22, 2015 4:51 PM, "Patrick Bergamin" < >>> patrick.bergamin at wedgenetworks.com> wrote: >>> >>>> I'm having problems with memory usage of the current implementation >>>> of EPollArraySelector in 1.8.0_60 for an existing proxy application. >>>> We've been on java version 1.7.0_05 for a while now because the new >>>> implementation of EPollArraySelector does not work well with the >>>> design of this proxy application. >>>> >>>> I did find the sun.nio.ch.maxUpdateArraySize property helped to >>>> reduce memory usage a bit. But as the proxy application runs >>>> all the EPollArraySelector objects will slowly accumulate memory. >>>> >>>> The current design of the proxy application is to have one Selector >>>> handle the listen socket. Once a connection is accepted, management >>>> of the connection is handed off to another thread on another Selector. >>>> Basically there is one Selector allocated per socket connection. >>>> The Selectors are never closed they are reused when a new socket >>>> connection is accepted. >>>> >>>> The application was designed before this latest implementation of >>>> EPollArraySelector. Redesigning the application to decouple the >>>> Selector from the socket connection would be a fair amount work. >>>> >>>> We have machines that are capable of running this proxy application >>>> with around 350,000 open connections. Since it is a proxy there >>>> are actually 700,000 open connections. The file descriptor limit >>>> is set high (5,000,000) to be able to handle all these open socket >>>> connections. >>>> >>>> Below I've included a patch to show what kinds of things I need >>>> to do to bring the memory usage of EPollArraySelector down for >>>> this proxy application. >>>> >>>> Is there any interest in including a second epoll implementation >>>> in openjdk that uses less memory or perhaps have more properties >>>> to control the memory usage of the existing EPollArraySelector? >>>> >>>> thanks, >>>> Patrick Bergamin >>>> >>>> --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 >>>> +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 -0600 >>>> @@ -29,6 +29,7 @@ >>>> import java.security.AccessController; >>>> import java.util.BitSet; >>>> import java.util.HashMap; >>>> +import java.util.HashSet; >>>> import java.util.Map; >>>> import sun.security.action.GetIntegerAction; >>>> >>>> @@ -122,7 +123,7 @@ >>>> >>>> // Used by release and updateRegistrations to track whether a file >>>> // descriptor is registered with epoll. >>>> - private final BitSet registered = new BitSet(); >>>> + private final HashSet registered = new HashSet(); >>>> >>>> >>>> EPollArrayWrapper() throws IOException { >>>> @@ -187,7 +188,10 @@ >>>> } >>>> } else { >>>> Integer key = Integer.valueOf(fd); >>>> - if (!isEventsHighKilled(key) || force) { >>>> + if (events == KILLED) { >>>> + eventsHigh.remove(key); >>>> + } >>>> + else if (!isEventsHighKilled(key) || force) { >>>> eventsHigh.put(key, Byte.valueOf(events)); >>>> } >>>> } >>>> @@ -201,6 +205,9 @@ >>>> return eventsLow[fd]; >>>> } else { >>>> Byte result = eventsHigh.get(Integer.valueOf(fd)); >>>> + if (result == null) { >>>> + return KILLED; >>>> + } >>>> // result should never be null >>>> return result.byteValue(); >>>> } >>>> @@ -235,7 +242,7 @@ >>>> // force the initial update events to 0 as it may be KILLED by >>>> a >>>> // previous registration. >>>> synchronized (updateLock) { >>>> - assert !registered.get(fd); >>>> + assert !registered.contains(fd); >>>> setUpdateEvents(fd, (byte)0, true); >>>> } >>>> } >>>> @@ -249,9 +256,9 @@ >>>> setUpdateEvents(fd, KILLED, false); >>>> >>>> // remove from epoll >>>> - if (registered.get(fd)) { >>>> + if (registered.contains(fd)) { >>>> epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); >>>> - registered.clear(fd); >>>> + registered.remove(fd); >>>> } >>>> } >>>> } >>>> @@ -286,7 +293,7 @@ >>>> while (j < updateCount) { >>>> int fd = updateDescriptors[j]; >>>> short events = getUpdateEvents(fd); >>>> - boolean isRegistered = registered.get(fd); >>>> + boolean isRegistered = registered.contains(fd); >>>> int opcode = 0; >>>> >>>> if (events != KILLED) { >>>> @@ -298,9 +305,9 @@ >>>> if (opcode != 0) { >>>> epollCtl(epfd, opcode, fd, events); >>>> if (opcode == EPOLL_CTL_ADD) { >>>> - registered.set(fd); >>>> + registered.add(fd); >>>> } else if (opcode == EPOLL_CTL_DEL) { >>>> - registered.clear(fd); >>>> + registered.remove(fd); >>>> } >>>> } >>>> } >>>> >>>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitalyd at gmail.com Tue Oct 27 20:34:32 2015 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Tue, 27 Oct 2015 16:34:32 -0400 Subject: Memory usage of EPollArraySelector In-Reply-To: References: <56294C20.4030300@wedgenetworks.com> <562A5B0B.4080708@wedgenetworks.com> <562A61E5.7020001@wedgenetworks.com> <562A6D1D.5060307@wedgenetworks.com> Message-ID: It's very common for epoll implementations to use a flat uint_32 array to map fd's. This makes it fast for the common case where there's either one or very few epoll loops. If you try to "address" this in the jdk, you'll need to make sure the common case lookups do not slow down. On Tue, Oct 27, 2015 at 4:31 PM, Martin Buchholz wrote: > I agree that the amount of memory retained in the jdk (linear in the file > descriptor limit) is a problem with the jdk that should be addressed > somehow. Is there an open bug? > > On Fri, Oct 23, 2015 at 10:23 AM, Patrick Bergamin < > patrick.bergamin at wedgenetworks.com> wrote: > >> Hi Vitaly, >> >> Yeah I understand what your saying. Right now we want to move forward >> with a newer version of java. This application is holding us back. >> Decoupling the Selectors from the socket connections is not a small task. >> All your suggestions are good ideas and match some of the ideas I have had >> on reworking the application. >> >> thanks for the help. >> >> >> On 15-10-23 10:45 AM, Vitaly Davidovich wrote: >> >> 80k worker threads?? That doesn't sound right either :). Roughly >> speaking, I suggest the following: >> >> 1) N compute bound worker threads where N = # of cpus >> 2) 1-2 i/o threads that monitor fd's for write/read readiness, and >> perform the read/write operations (workers in #1 hand off data to these) >> 3) Some threadpool for IO/blocking operations where you don't have async >> options (e.g. filesystem/disk) which you can size depending on latency of >> the i/o operations >> >> Do some research on modern i/o threading models (e.g. nginx, netty, >> etc). It may be a larger effort, but you'll be happier in the long run. >> >> >> >> On Fri, Oct 23, 2015 at 12:35 PM, Patrick Bergamin < >> patrick.bergamin at wedgenetworks.com> >> wrote: >> >>> I thought about using one Selector per thread. I wasn't sure if that >>> was going reduce memory usage enough though as the application can allow >>> upwards of 80000 worker threads. I should try this out as it isn't a large >>> change to the application. >>> >>> thanks. >>> >>> >>> >>> On 15-10-23 10:23 AM, Vitaly Davidovich wrote: >>> >>> The entire problem is you have a Selector per connection rather than >>> Selector per worker thread, as I mentioned in my previous reply. I don't >>> think Selector was designed for such a thing, and so using a BitSet makes >>> sense for the intended usage. HashSet will reduce your memory footprint >>> because you'll have just 1 entry in there, but the common/intended case >>> will make it worse as it has terrible memory locality properties. Your >>> real fix is to redesign the Selector usage within your application. >>> >>> On Fri, Oct 23, 2015 at 12:06 PM, Patrick Bergamin < >>> patrick.bergamin at wedgenetworks.com> >>> wrote: >>> >>>> On 15-10-22 03:39 PM, Vitaly Davidovich wrote: >>>> >>>> Patrick, >>>> >>>> >>>> Yes Selectors are reused. One Selector is used per connection. When a >>>> socket connection is closed the Selectors are cached along with a >>>> connection handler object so they can be reused when new socket connections >>>> are accepted. >>>> >>>> I'm confused - how many selectors do you have? You mentioned a selector >>>> is created for each accepted connection but then state that selectors are >>>> reused for newly accepted connections. Do you mean a selector is reused >>>> after its sole connection is closed? >>>> >>>> >>>> I'm not sure what you mean by memory chasing. I changed the BitSet to >>>> a HashSet because the file descriptor numbers can get easily get into the >>>> millions. BitSet can allocate a good chunk of memory even if only a couple >>>> of file descriptors are registered with it (if the file descriptor numbers >>>> are large). When you have hundreds of thousands of open Selectors the >>>> memory usage adds up. This is one reason why simply closing Selectors >>>> after a connection is closed does not solve the memory usage problem. >>>> >>>> From my perspective it is the EpollArrayWrapper that is wasting memory >>>> resources. It can allocate memory that is not used and never releases any >>>> that is allocated. Likely this was done for performance reasons. I >>>> haven't looked back through the revision history. >>>> >>>> I'm not proposing the suggested diff to be committed but rather I'm >>>> presenting it to show where the memory is accumulating in the application. >>>> After sun.nio.ch.maxUpdateArraySize was set to a smaller value it was the >>>> 'registered' object and 'eventsHigh' object within EpollArrayWrapper that >>>> were accumulating memory. I'm asking if there is something that can be >>>> done to reduce the memory usage of EpollArrayWrapper by either adding more >>>> system properties to change its behaviour or create a second implementation? >>>> >>>> thanks, >>>> Patrick Bergamin >>>> >>>> >>>> You should bite the bullet and reimplement your application. If I'm >>>> reading right, you're wasting lots of resources all around, and your >>>> proposal is just a hack (with adverse performance due to additional memory >>>> chasing via HashSet) that will likely catch up to you anyway. >>>> >>>> sent from my phone >>>> On Oct 22, 2015 4:51 PM, "Patrick Bergamin" < >>>> patrick.bergamin at wedgenetworks.com> wrote: >>>> >>>>> I'm having problems with memory usage of the current implementation >>>>> of EPollArraySelector in 1.8.0_60 for an existing proxy application. >>>>> We've been on java version 1.7.0_05 for a while now because the new >>>>> implementation of EPollArraySelector does not work well with the >>>>> design of this proxy application. >>>>> >>>>> I did find the sun.nio.ch.maxUpdateArraySize property helped to >>>>> reduce memory usage a bit. But as the proxy application runs >>>>> all the EPollArraySelector objects will slowly accumulate memory. >>>>> >>>>> The current design of the proxy application is to have one Selector >>>>> handle the listen socket. Once a connection is accepted, management >>>>> of the connection is handed off to another thread on another Selector. >>>>> Basically there is one Selector allocated per socket connection. >>>>> The Selectors are never closed they are reused when a new socket >>>>> connection is accepted. >>>>> >>>>> The application was designed before this latest implementation of >>>>> EPollArraySelector. Redesigning the application to decouple the >>>>> Selector from the socket connection would be a fair amount work. >>>>> >>>>> We have machines that are capable of running this proxy application >>>>> with around 350,000 open connections. Since it is a proxy there >>>>> are actually 700,000 open connections. The file descriptor limit >>>>> is set high (5,000,000) to be able to handle all these open socket >>>>> connections. >>>>> >>>>> Below I've included a patch to show what kinds of things I need >>>>> to do to bring the memory usage of EPollArraySelector down for >>>>> this proxy application. >>>>> >>>>> Is there any interest in including a second epoll implementation >>>>> in openjdk that uses less memory or perhaps have more properties >>>>> to control the memory usage of the existing EPollArraySelector? >>>>> >>>>> thanks, >>>>> Patrick Bergamin >>>>> >>>>> --- EPollArrayWrapper.java 2015-07-30 06:27:02.000000000 -0600 >>>>> +++ EPollArrayWrapper2.java 2015-09-28 15:31:41.712607415 -0600 >>>>> @@ -29,6 +29,7 @@ >>>>> import java.security.AccessController; >>>>> import java.util.BitSet; >>>>> import java.util.HashMap; >>>>> +import java.util.HashSet; >>>>> import java.util.Map; >>>>> import sun.security.action.GetIntegerAction; >>>>> >>>>> @@ -122,7 +123,7 @@ >>>>> >>>>> // Used by release and updateRegistrations to track whether a file >>>>> // descriptor is registered with epoll. >>>>> - private final BitSet registered = new BitSet(); >>>>> + private final HashSet registered = new >>>>> HashSet(); >>>>> >>>>> >>>>> EPollArrayWrapper() throws IOException { >>>>> @@ -187,7 +188,10 @@ >>>>> } >>>>> } else { >>>>> Integer key = Integer.valueOf(fd); >>>>> - if (!isEventsHighKilled(key) || force) { >>>>> + if (events == KILLED) { >>>>> + eventsHigh.remove(key); >>>>> + } >>>>> + else if (!isEventsHighKilled(key) || force) { >>>>> eventsHigh.put(key, Byte.valueOf(events)); >>>>> } >>>>> } >>>>> @@ -201,6 +205,9 @@ >>>>> return eventsLow[fd]; >>>>> } else { >>>>> Byte result = eventsHigh.get(Integer.valueOf(fd)); >>>>> + if (result == null) { >>>>> + return KILLED; >>>>> + } >>>>> // result should never be null >>>>> return result.byteValue(); >>>>> } >>>>> @@ -235,7 +242,7 @@ >>>>> // force the initial update events to 0 as it may be KILLED >>>>> by a >>>>> // previous registration. >>>>> synchronized (updateLock) { >>>>> - assert !registered.get(fd); >>>>> + assert !registered.contains(fd); >>>>> setUpdateEvents(fd, (byte)0, true); >>>>> } >>>>> } >>>>> @@ -249,9 +256,9 @@ >>>>> setUpdateEvents(fd, KILLED, false); >>>>> >>>>> // remove from epoll >>>>> - if (registered.get(fd)) { >>>>> + if (registered.contains(fd)) { >>>>> epollCtl(epfd, EPOLL_CTL_DEL, fd, 0); >>>>> - registered.clear(fd); >>>>> + registered.remove(fd); >>>>> } >>>>> } >>>>> } >>>>> @@ -286,7 +293,7 @@ >>>>> while (j < updateCount) { >>>>> int fd = updateDescriptors[j]; >>>>> short events = getUpdateEvents(fd); >>>>> - boolean isRegistered = registered.get(fd); >>>>> + boolean isRegistered = registered.contains(fd); >>>>> int opcode = 0; >>>>> >>>>> if (events != KILLED) { >>>>> @@ -298,9 +305,9 @@ >>>>> if (opcode != 0) { >>>>> epollCtl(epfd, opcode, fd, events); >>>>> if (opcode == EPOLL_CTL_ADD) { >>>>> - registered.set(fd); >>>>> + registered.add(fd); >>>>> } else if (opcode == EPOLL_CTL_DEL) { >>>>> - registered.clear(fd); >>>>> + registered.remove(fd); >>>>> } >>>>> } >>>>> } >>>>> >>>>> >>>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: