JDK-8009302 (infinite recursion on AppKit thread / stack overflow)

[Gerard Ziemski]

hi guys,

After learning about signals, stacks and threads I found one possible 
solution that involves setting up an alternate stack (sigaltstack) and 
asking signals to use it (SA_ONSTACK).

This would require us to modify 2 files:

- os_bsd.cpp, method os::Bsd::set_signal_handler needs:

#if __APPLE__
     // needed by AppKit thread
     if (sig == SIGSEGV) {
         sigAct.sa_flags |= SA_ONSTACK;
     }
#endif

- jni.cpp , method attach_current_thread (though it should probably live 
somewhere in thread.cpp instead) needs:

#if __APPLE__
     // create alternate stack for SIGSEGV
     stack_t sigstack = {0};
     sigstack.ss_flags = 0;
     sigstack.ss_size = SIGSTKSZ;
     sigstack.ss_sp = valloc(sigstack.ss_size); // page aligned memory
     if (sigstack.ss_sp != NULL) {
         if (sigaltstack(&sigstack, NULL) == -1) {
             fprintf(stderr, "sigaltstack err\n");
         }
     } else {
         fprintf(stderr, "valloc err\n");
     }
#endif

This would require us to increase slightly memory per app as the 
alternate stack requires some extra memory (SIGSTKSZ, ie. 128K on Mac), 
though that value could be lowered down possibly.

Another advantage here is that if recursion happens even somewhere in 
native code, it will also be caught and proper Java dump stack produced, 
as opposed to Mac OS X CrashReporter popping up.

During this morning meeting it was mentioned that we used to create 
alternate stack, but that it wasn't reliable.

What exactly was the problem with alt stack? Was it for a specific platform?

The fix here would only apply to Mac OS X.

It's possible we can use XNU instead (just like 
http://www.gnu.org/software/libsigsegv/) to catch the kernel level 
signal and make sure our POSIX signal handler gets it, but what about 
stack corruption if the thread that gets it was the one that had its 
stack corrupted? (not sure how XNU handles this - I will look into this) 
Seems we must have a guaranteed clean stack somehow in this case?


I have attached the updated test cases and source code (the RecursionC 
and Signals should be platform independent - though they have Xcode 
projects provided, RecursionAppkit requires Mac and Xcode)

BTW If someone could check out my "Signals" code and tell me why the 
signals seem to be ignored with dedicated signal thread, that would be 
great.

https://jbs.oracle.com/bugs/browse/JDK-8009302


cheers