RFC: 8216472: Stack overflow followed by crash

[Alex Kashchenko]

Hi,

At Red Hat, we've got a windows-only crash in nio.dll with a JBoss 
application,  that appeared to be JDK-8216472 [1].

After some investigation it was found, that 
Java_sun_nio_ch_WindowsSelectorImpl_00024SubSelector_poll0 function [2] 
allocates more than 50Kb on stack - because FD_SETSIZE is set to 1024 
[3], each fd_set structure takes 8200 bytes and 6 fd_set structures are 
placed on stack [4][5]. If this function is called from a deep recursive 
java call, it can cause stack overflow and JVM process is either aborted 
or killed by OS.

One of the solutions to this may be to set hotspot StackShadowPages 
parameter to 14. Currently it is 7 on windows and 20 on linux.

I'd like to solicit comments and suggestions about another solution, 
that changes this native call in NIO moving fd_sets to heap:

http://cr.openjdk.java.net/~akasko/jdk/8216472/webrev.00/

PollOverflow reproducer there causes JVM abort (or silent kill) without 
the patch.

We tested this patch internally and haven't found any regressions; tests 
included JCK and SPECjbb, though we haven't run any benchmarks 
specifically oriented on networking performance.


[1] https://bugs.openjdk.java.net/browse/JDK-8216472
[2] 
https://hg.openjdk.java.net/jdk/jdk/file/7f34de3cdfe9/src/java.base/windows/native/libnio/ch/WindowsSelectorImpl.c#l55
[3] 
https://hg.openjdk.java.net/jdk/jdk/file/7f34de3cdfe9/src/java.base/windows/native/libnio/ch/WindowsSelectorImpl.c#l33
[4] 
https://hg.openjdk.java.net/jdk/jdk/file/7f34de3cdfe9/src/java.base/windows/native/libnio/ch/WindowsSelectorImpl.c#l64
[5] 
https://hg.openjdk.java.net/jdk/jdk/file/7f34de3cdfe9/src/java.base/windows/native/libnio/ch/WindowsSelectorImpl.c#l117

-- 
-Alex