Regression in EPollArrayWrapper causes NPE when fd > 64 * 1024

[Norman Maurer]

Hi there,

this is my first bug-report here so bear with me if something is missing ;)

During testing netty.io with many concurrent connections one of our users reported a NullPointerException which was thrown by sun.nio.ch.EPollArrayWrapper.setUpdateEvents(...). This was observed as soon as the concurrent connection count > 64 * 1024. After more investigating I was able to find the bug in EPollArrayWrapper.setUpdateEvents(...), which is a regression introduced by the following change:

http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/rev/017bd924a3c8

The problem here is that eventsHigh.get(key) will be called once the fd is > 64 * 1024. This may return "null" which is compared to KILLED (which is of type byte) and so may throw a NPE because the compare tries to unbox the return value (which is of type Byte). The regression is present in lastest openjdk8 and in openjdk7u40 and later. It seems to also affects oracle jdk 7u40. All OS'es that use epoll are affected, in my case linux (ubuntu).

Attached you find the proposed fix for openjdk8 and openjdk7 and a reproducer which can be used. The fix does two things:
* Eliminate the access to  the eventsHigh Map if "force" is true.
* Check for null before try to compare the stored events value


Reproducer:
The reproducer will bind a server to the specified port and just accept new connections. The clients will connect to the server until 80 * 1024 connections are reached, and then go to sleep. When you use the reproducer without the attached fix the server will fail with:

Exception in thread "main" java.lang.NullPointerException
	at sun.nio.ch.EPollArrayWrapper.setUpdateEvents(EPollArrayWrapper.java:178)
	at sun.nio.ch.EPollArrayWrapper.add(EPollArrayWrapper.java:227)
	at sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:164)
	at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:132)
	at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212)
	at java.nio.channels.SelectableChannel.register(SelectableChannel.java:280)
	at BugReproducer.main(BugReproducer.java:39)

Once the fix is applied no Exception is thrown anymore. 

To run the reproducer you may need to update the localport range. For this use the following command:
# sudo sysctl -w net.ipv4.ip_local_port_range="1024 64000"

Also you may need to increate ulimit to something big enough. I'm using 1048576 here to test it.

You need two interfaces if you not have more then one you can use a virtual interface. So if you have for example eth0 with the ipaddress 10.0.0.9 you can use something like the following to setup a virtual interface:
# sudo ifconfig eth0:1 10.0.0.10 broadcast 255.255.255.0

Compile the reproducer classes, which you should put in somedir (here we call it /tmp/niobug/)
# cd /tmp/niobug/
# javac -cp . Bug*.class

After the classes are compiled you will need 3 terminals. One for the server and the other 2 to run the clients.
First Terminal:
# java -Djava.net.preferIPv4Stack=true -cp . BugReproducer 8080

Second Terminal:
# java -Djava.net.preferIPv4Stack=true -cp . BugReproducerClient 10.0.0.9 8080

Third Terminal:
# java -Djava.net.preferIPv4Stack=true -cp . BugReproducerClient 10.0.0.10 8080

You will see the concurrent connection count in the First Terminal.






Thanks,
Norman


---
Norman Maurer
nmaurer at redhat.com

JBoss, by Red Hat



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: BugReproducer.java
Type: application/octet-stream
Size: 1753 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/BugReproducer.java 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: BugReproducerClient.java
Type: application/octet-stream
Size: 1927 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/BugReproducerClient.java 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/attachment-0002.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: openjdk7-epollarraywrapper-fix.patch
Type: application/octet-stream
Size: 1238 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/openjdk7-epollarraywrapper-fix.patch 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/attachment-0003.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: openjdk8-epollarraywrapper-fix.patch
Type: application/octet-stream
Size: 1238 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/openjdk8-epollarraywrapper-fix.patch 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/nio-dev/attachments/20130916/9ff39ec1/attachment-0004.html