jstack can't attach to JVM (OS X)

[Mikael Gerdin]

On 2013-08-27 10:26, Staffan Larsen wrote:
>
> On 26 aug 2013, at 20:19, Martin Traverso <mtraverso at gmail.com
> <mailto:mtraverso at gmail.com>> wrote:
>
>> Great! Thanks for the explanation.
>>
>> Just curious, why does running jstack at least once before the code
>> reaches that method call prevent the problem from happening at all?
>
> No idea. :(  Can't figure it out.

I think SIGQUIT is only used to initiate the attach mechanism. If the 
JVM can catch the SIGQUIT before the verification error has occurred 
then the attach mechanism will have been initialized and subsequent 
jstack/jcmd commands can be processed.

/Mikael

>
> /Staffan
>
>>
>> Martin
>>
>>
>> On Mon, Aug 26, 2013 at 5:31 AM, Staffan Larsen
>> <staffan.larsen at oracle.com <mailto:staffan.larsen at oracle.com>> wrote:
>>
>>     Here is what's happening:
>>
>>     - The VM sets the signal mask for all threads (except the internal
>>     VMThread) to mask out SIGQUIT using pthread_sigmask(). So the
>>     process will still respond to SIGQUIT, but only one thread.
>>     - The verifier code calls setjmp() to save the calling context. On
>>     OS X this also saves the signal mask.
>>     - The example code causes a verification error somewhere which
>>     causes the verifier to call longjmp().
>>     - longjmp will restore the signal mask using sigprocmask() which
>>     sets the signal mask for the _process_.
>>     - Now the process has a signal mask that masks out SIGQUIT and
>>     attach stops working.
>>
>>     This only happens on OS X because setjmp/longjmp does not save and
>>     restore the signal mask on other platforms. There are functions
>>     _setjmp/_longjmp on OS X to skip the signal mask save/restore and
>>     I think this is what we need to use in the verifier (also need to
>>     check other uses of setjmp/longjmp in the JDK).
>>
>>     I have filed bug 8023720 for this.
>>
>>     Thanks again for the report and reproducer.
>>
>>     /Staffan
>>
>>
>>
>>
>>     On 23 aug 2013, at 14:44, Staffan Larsen
>>     <staffan.larsen at oracle.com <mailto:staffan.larsen at oracle.com>> wrote:
>>
>>>     Thanks for the excellent bugreport!
>>>
>>>     I've had a quick look today but haven't yet figured out what the
>>>     problem is. What happens is that somehow the process becomes
>>>     immune to the SIGQUIT that jstack sends to it when it wants to
>>>     attach. Instead of running jstack, one can also do "kill -QUIT
>>>     <pid>" and with this code no thread stacks are printed (as they
>>>     normally will be).
>>>
>>>     I'll continue looking,
>>>     /Staffan
>>>
>>>     On 23 aug 2013, at 07:20, Martin Traverso <mtraverso at gmail.com
>>>     <mailto:mtraverso at gmail.com>> wrote:
>>>
>>>>     I have a program that causes jstack and jcmd to fail to attach
>>>>     to the JVM on OS X. Unfortunately, it uses a third-party
>>>>     library, so I'm not sure what's the magic incantation that
>>>>     causes this to happen.
>>>>
>>>>     I wrote a small test case using this library that reproduces the
>>>>     problem. You can find the code here:
>>>>     https://github.com/martint/jstackissue.
>>>>
>>>>     I've seen the problem on both 1.7.0_25-b15 and 1.8.0-ea-b100 on
>>>>     OS X, but not on Java 6. It doesn't seem to happen on Linux, either.
>>>>
>>>>     One interesting data point is that if I run jstack before the
>>>>     code makes the call to the library, it works, and continues to
>>>>     work even after that call is complete. On the other hand, if run
>>>>     jstack for the first time *after* the call to the library, it
>>>>     won't work at all.
>>>>
>>>>     Any ideas?
>>>>
>>>>     Thanks!
>>>>     Martin
>>>
>>
>>
>