jstack and the target output

[Piotr Bzdyl]

After checking the strace output I notice that the issue was caused by the
file mode for files created on vboxsf mount. By default (when automount
option is used in virtualbox configuration) it creates .attach_pid<pid>
with file mode 0770. When I manually mounted shared folder specifying
fmode=660 jstack worked as expected.

Thank you for pointing me to the right path to narrow down the issue.

BTW, where I could find the JVM code responsible for handling SIGQUIT
signal to see what it does that file mode matters?



On Fri, Nov 22, 2013 at 10:30 AM, Piotr Bzdyl <piotr at bzdyl.net> wrote:

> Thank you Mikael for the detailed information - I learned a lot about how
> SA tools attache to the target process.
>
> When I was running the test you suggested I noticed one "detail" in my
> setup which turned out to be the cause of my problems: as I mentioned
> previously I was running my tests inside a virtual machine (namely
> VirtualBox). The issue was that the cwd dir of my target process was on the
> mounted shared directory (vboxsf filesystem) in VirtualBox VM and it was
> causing the problems. I still have to go through the strace outputs to find
> out if it is because of the access rights of because how vboxsf file system
> driver works (attached the output).
>
>
> On Fri, Nov 22, 2013 at 9:51 AM, Mikael Gerdin <mikael.gerdin at oracle.com>wrote:
>
>> Piotr,
>>
>> On Friday 22 November 2013 09.21.56 Piotr Bzdyl wrote:
>> > Is there any other information I could collect to help troubleshoot this
>> > problem? How could I check if VM runs with attach disabled?
>>
>> The comments in
>>
>> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java
>> describe the handshaking involved in dynamically enabling the attach
>> mechanism
>> on Linux.
>>
>> The mechanism is basically:
>> * The attacher checks for the existence of the socket file
>> /tmp/.java_pid${PID}
>> * If that does not exist it creates a file .attach_pid${PID} in the target
>> process' working directory and sends a SIGQUIT (kill -QUIT) to that
>> process.
>> * The SIGQUIT handler notices the .attach_pid${PID} file and creates the
>> socket at /tmp/.java_pid${PID}
>> * The attacher notices the socket file and unlinks the .attach_pid${PID}
>> file.
>>
>> You can force the target process to always create the socket file at
>> startup
>> by using the -XX:+StartAttachListener command line flag.
>>
>> To further troubleshoot the problem I suggest that you try strace:ing the
>> target process and jstack with the following flags enabled:
>>
>> (target process)
>> $ strace -f -e trace=accept,stat,socket,bind $JAVA_HOME/bin/java donothing
>>
>> (jstack process)
>> $ strace -f -e trace=open,connect,stat,unlink,write,kill
>> $JAVA_HOME/bin/jstack
>> <pid>
>>
>> You should get something similar to the following:
>> (target process)
>> (...)
>> [pid 12602] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=12687,
>> si_uid=726} ---
>> [pid 12630] stat(".attach_pid12602", {st_mode=S_IFREG|0644, st_size=0,
>> ...}) =
>> 0
>> Process 12733 attached
>> [pid 12733] socket(PF_LOCAL, SOCK_STREAM, 0) = 5
>> [pid 12733] bind(5, {sa_family=AF_LOCAL,
>> sun_path="/tmp/.java_pid12602.tmp"},
>> 110) = 0
>> [pid 12733] accept(5, {sa_family=AF_LOCAL, NULL}, [2]) = 6
>>
>> (jstack process)
>> [pid 12688] stat("/localhome/java/jdk-8-ea-bin-
>> b112/jre/lib/amd64/libattach.so", {st_mode=S_IFREG|0755, st_size=16029,
>> ...})
>> = 0
>> [pid 12688] open("/localhome/java/jdk-8-ea-bin-
>> b112/jre/lib/amd64/libattach.so", O_RDONLY) = 7
>> [pid 12688] open("/localhome/java/jdk-8-ea-bin-
>> b112/jre/lib/amd64/libattach.so", O_RDONLY|O_CLOEXEC) = 7
>> [pid 12688] stat("/tmp/.java_pid12602", 0x7fe53183e3a0) = -1 ENOENT (No
>> such
>> file or directory)
>> [pid 12688] open("/proc/12602/cwd/.attach_pid12602",
>> O_RDWR|O_CREAT|O_EXCL,
>> 0666) = 7
>> [pid 12688] kill(12602, SIGQUIT)        = 0
>> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, st_size=0,
>> ...}) = 0
>> [pid 12688] unlink("/proc/12602/cwd/.attach_pid12602") = 0
>> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, st_size=0,
>> ...}) = 0
>> [pid 12688] connect(7, {sa_family=AF_LOCAL,
>> sun_path="/tmp/.java_pid12602"},
>> 110) = 0
>> [pid 12688] connect(7, {sa_family=AF_LOCAL,
>> sun_path="/tmp/.java_pid12602"},
>> 110) = 0
>> [pid 12688] write(7, "1", 1)            = 1
>> [pid 12688] write(7, "\0", 1)           = 1
>> [pid 12688] write(7, "threaddump", 10)  = 10
>> [pid 12688] write(7, "\0", 1)           = 1
>>
>> /Mikael
>>
>> >
>> > I noticed the same behavior when trying to connect to the process using
>> > JConsole: JConsole displays the sample app in the list and when I try to
>> > connect it finally displays an error that it cannot connect whereas the
>> > sample app prints thread dump on its console.
>> >
>> > On Thu, Nov 21, 2013 at 7:18 PM, Alan Bateman
>> <Alan.Bateman at oracle.com>wrote:
>> > > On 21/11/2013 14:47, David Holmes wrote:
>> > >> I can recreate this using the minimal VM, which doesn't support
>> > >> libattach. So it definitely seems like something is going wrong in
>> the
>> > >> attach process, but I can't say what. Not sure what debugging hooks
>> we
>> > >> have
>> > >> for the attach mechanism ??
>> > >
>> > > The first attach involves signaling the target VM to start up the
>> attach
>> > > mechanism. So for the minimal VM or a VM running with attach disabled
>> (or
>> > > as Staffan pointed out, running with a different tmp directory) then
>> the
>> > > target VM gets a SIGQUIT and just prints the thread dump. It's hard
>> to say
>> > > what Piotr is running into but I think this at least explains it for
>> the
>> > > minimal VM.
>> > >
>> > > -Alan
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20131122/c40f9840/attachment.html