jstack can't attach to JVM (OS X)

[David Holmes]

On 26/08/2013 10:31 PM, Staffan Larsen wrote:
> Here is what's happening:
>
> - The VM sets the signal mask for all threads (except the internal
> VMThread) to mask out SIGQUIT using pthread_sigmask(). So the process
> will still respond to SIGQUIT, but only one thread.
> - The verifier code calls setjmp() to save the calling context. On OS X
> this also saves the signal mask.
> - The example code causes a verification error somewhere which causes
> the verifier to call longjmp().
> - longjmp will restore the signal mask using sigprocmask() which sets
> the signal mask for the _process_.

Wow! Talk about broken! :( If you are going to provide system calls that 
look like POSIX then they should implement POSIX semantics!

"The use of the sigprocmask() function is unspecified in a 
multi-threaded process."

http://pubs.opengroup.org/onlinepubs/9699919799/functions/sigprocmask.html

And guess what:

https://developer.apple.com/library/ios/documentation/system/conceptual/manpages_iphoneos/man2/sigprocmask.2.html#//apple_ref/doc/man/2/sigprocmask

    The sigprocmask() function call is expected to conform to IEEE Std
      1003.1-1988 (``POSIX.1'').

Maybe they need to bring their operating system into the 21st century! 
Sheesh!

David
-----

> - Now the process has a signal mask that masks out SIGQUIT and attach
> stops working.
>
> This only happens on OS X because setjmp/longjmp does not save and
> restore the signal mask on other platforms. There are functions
> _setjmp/_longjmp on OS X to skip the signal mask save/restore and I
> think this is what we need to use in the verifier (also need to check
> other uses of setjmp/longjmp in the JDK).
>
> I have filed bug 8023720 for this.
>
> Thanks again for the report and reproducer.
>
> /Staffan
>
>
>
>
> On 23 aug 2013, at 14:44, Staffan Larsen <staffan.larsen at oracle.com
> <mailto:staffan.larsen at oracle.com>> wrote:
>
>> Thanks for the excellent bugreport!
>>
>> I've had a quick look today but haven't yet figured out what the
>> problem is. What happens is that somehow the process becomes immune to
>> the SIGQUIT that jstack sends to it when it wants to attach. Instead
>> of running jstack, one can also do "kill -QUIT <pid>" and with this
>> code no thread stacks are printed (as they normally will be).
>>
>> I'll continue looking,
>> /Staffan
>>
>> On 23 aug 2013, at 07:20, Martin Traverso <mtraverso at gmail.com
>> <mailto:mtraverso at gmail.com>> wrote:
>>
>>> I have a program that causes jstack and jcmd to fail to attach to the
>>> JVM on OS X. Unfortunately, it uses a third-party library, so I'm not
>>> sure what's the magic incantation that causes this to happen.
>>>
>>> I wrote a small test case using this library that reproduces the
>>> problem. You can find the code here:
>>> https://github.com/martint/jstackissue.
>>>
>>> I've seen the problem on both 1.7.0_25-b15 and 1.8.0-ea-b100 on OS X,
>>> but not on Java 6. It doesn't seem to happen on Linux, either.
>>>
>>> One interesting data point is that if I run jstack before the code
>>> makes the call to the library, it works, and continues to work even
>>> after that call is complete. On the other hand, if run jstack for the
>>> first time *after* the call to the library, it won't work at all.
>>>
>>> Any ideas?
>>>
>>> Thanks!
>>> Martin
>>
>