JDK-8009302 (infinite recursion on AppKit thread / stack overflow)

[David Holmes]

Hi Gerard,

Switching to alt-signal-stack is not a trivial change and has been 
discussed before (please search archives).

As I wrote in the CR I still don't see anything showing exactly why the 
problem is arising. I'm less interested at this stage in what avoids the 
problem, than what causes the problem: why is the stack of the appkit 
thread laid out differently and what is that layout?

Cheers,
David

On 10/04/2013 5:56 AM, Gerard Ziemski wrote:
> hi guys,
>
> After learning about signals, stacks and threads I found one possible
> solution that involves setting up an alternate stack (sigaltstack) and
> asking signals to use it (SA_ONSTACK).
>
> This would require us to modify 2 files:
>
> - os_bsd.cpp, method os::Bsd::set_signal_handler needs:
>
> #if __APPLE__
>      // needed by AppKit thread
>      if (sig == SIGSEGV) {
>          sigAct.sa_flags |= SA_ONSTACK;
>      }
> #endif
>
> - jni.cpp , method attach_current_thread (though it should probably live
> somewhere in thread.cpp instead) needs:
>
> #if __APPLE__
>      // create alternate stack for SIGSEGV
>      stack_t sigstack = {0};
>      sigstack.ss_flags = 0;
>      sigstack.ss_size = SIGSTKSZ;
>      sigstack.ss_sp = valloc(sigstack.ss_size); // page aligned memory
>      if (sigstack.ss_sp != NULL) {
>          if (sigaltstack(&sigstack, NULL) == -1) {
>              fprintf(stderr, "sigaltstack err\n");
>          }
>      } else {
>          fprintf(stderr, "valloc err\n");
>      }
> #endif
>
> This would require us to increase slightly memory per app as the
> alternate stack requires some extra memory (SIGSTKSZ, ie. 128K on Mac),
> though that value could be lowered down possibly.
>
> Another advantage here is that if recursion happens even somewhere in
> native code, it will also be caught and proper Java dump stack produced,
> as opposed to Mac OS X CrashReporter popping up.
>
> During this morning meeting it was mentioned that we used to create
> alternate stack, but that it wasn't reliable.
>
> What exactly was the problem with alt stack? Was it for a specific
> platform?
>
> The fix here would only apply to Mac OS X.
>
> It's possible we can use XNU instead (just like
> http://www.gnu.org/software/libsigsegv/) to catch the kernel level
> signal and make sure our POSIX signal handler gets it, but what about
> stack corruption if the thread that gets it was the one that had its
> stack corrupted? (not sure how XNU handles this - I will look into this)
> Seems we must have a guaranteed clean stack somehow in this case?
>
>
> I have attached the updated test cases and source code (the RecursionC
> and Signals should be platform independent - though they have Xcode
> projects provided, RecursionAppkit requires Mac and Xcode)
>
> BTW If someone could check out my "Signals" code and tell me why the
> signals seem to be ignored with dedicated signal thread, that would be
> great.
>
> https://jbs.oracle.com/bugs/browse/JDK-8009302
>
>
> cheers