jstack and the target output

[Piotr Bzdyl]

Thank you Mikael for the detailed information - I learned a lot about how
SA tools attache to the target process.

When I was running the test you suggested I noticed one "detail" in my
setup which turned out to be the cause of my problems: as I mentioned
previously I was running my tests inside a virtual machine (namely
VirtualBox). The issue was that the cwd dir of my target process was on the
mounted shared directory (vboxsf filesystem) in VirtualBox VM and it was
causing the problems. I still have to go through the strace outputs to find
out if it is because of the access rights of because how vboxsf file system
driver works (attached the output).


On Fri, Nov 22, 2013 at 9:51 AM, Mikael Gerdin <mikael.gerdin at oracle.com>wrote:

> Piotr,
>
> On Friday 22 November 2013 09.21.56 Piotr Bzdyl wrote:
> > Is there any other information I could collect to help troubleshoot this
> > problem? How could I check if VM runs with attach disabled?
>
> The comments in
>
> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java
> describe the handshaking involved in dynamically enabling the attach
> mechanism
> on Linux.
>
> The mechanism is basically:
> * The attacher checks for the existence of the socket file
> /tmp/.java_pid${PID}
> * If that does not exist it creates a file .attach_pid${PID} in the target
> process' working directory and sends a SIGQUIT (kill -QUIT) to that
> process.
> * The SIGQUIT handler notices the .attach_pid${PID} file and creates the
> socket at /tmp/.java_pid${PID}
> * The attacher notices the socket file and unlinks the .attach_pid${PID}
> file.
>
> You can force the target process to always create the socket file at
> startup
> by using the -XX:+StartAttachListener command line flag.
>
> To further troubleshoot the problem I suggest that you try strace:ing the
> target process and jstack with the following flags enabled:
>
> (target process)
> $ strace -f -e trace=accept,stat,socket,bind $JAVA_HOME/bin/java donothing
>
> (jstack process)
> $ strace -f -e trace=open,connect,stat,unlink,write,kill
> $JAVA_HOME/bin/jstack
> <pid>
>
> You should get something similar to the following:
> (target process)
> (...)
> [pid 12602] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=12687,
> si_uid=726} ---
> [pid 12630] stat(".attach_pid12602", {st_mode=S_IFREG|0644, st_size=0,
> ...}) =
> 0
> Process 12733 attached
> [pid 12733] socket(PF_LOCAL, SOCK_STREAM, 0) = 5
> [pid 12733] bind(5, {sa_family=AF_LOCAL,
> sun_path="/tmp/.java_pid12602.tmp"},
> 110) = 0
> [pid 12733] accept(5, {sa_family=AF_LOCAL, NULL}, [2]) = 6
>
> (jstack process)
> [pid 12688] stat("/localhome/java/jdk-8-ea-bin-
> b112/jre/lib/amd64/libattach.so", {st_mode=S_IFREG|0755, st_size=16029,
> ...})
> = 0
> [pid 12688] open("/localhome/java/jdk-8-ea-bin-
> b112/jre/lib/amd64/libattach.so", O_RDONLY) = 7
> [pid 12688] open("/localhome/java/jdk-8-ea-bin-
> b112/jre/lib/amd64/libattach.so", O_RDONLY|O_CLOEXEC) = 7
> [pid 12688] stat("/tmp/.java_pid12602", 0x7fe53183e3a0) = -1 ENOENT (No
> such
> file or directory)
> [pid 12688] open("/proc/12602/cwd/.attach_pid12602", O_RDWR|O_CREAT|O_EXCL,
> 0666) = 7
> [pid 12688] kill(12602, SIGQUIT)        = 0
> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, st_size=0,
> ...}) = 0
> [pid 12688] unlink("/proc/12602/cwd/.attach_pid12602") = 0
> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, st_size=0,
> ...}) = 0
> [pid 12688] connect(7, {sa_family=AF_LOCAL,
> sun_path="/tmp/.java_pid12602"},
> 110) = 0
> [pid 12688] connect(7, {sa_family=AF_LOCAL,
> sun_path="/tmp/.java_pid12602"},
> 110) = 0
> [pid 12688] write(7, "1", 1)            = 1
> [pid 12688] write(7, "\0", 1)           = 1
> [pid 12688] write(7, "threaddump", 10)  = 10
> [pid 12688] write(7, "\0", 1)           = 1
>
> /Mikael
>
> >
> > I noticed the same behavior when trying to connect to the process using
> > JConsole: JConsole displays the sample app in the list and when I try to
> > connect it finally displays an error that it cannot connect whereas the
> > sample app prints thread dump on its console.
> >
> > On Thu, Nov 21, 2013 at 7:18 PM, Alan Bateman
> <Alan.Bateman at oracle.com>wrote:
> > > On 21/11/2013 14:47, David Holmes wrote:
> > >> I can recreate this using the minimal VM, which doesn't support
> > >> libattach. So it definitely seems like something is going wrong in the
> > >> attach process, but I can't say what. Not sure what debugging hooks we
> > >> have
> > >> for the attach mechanism ??
> > >
> > > The first attach involves signaling the target VM to start up the
> attach
> > > mechanism. So for the minimal VM or a VM running with attach disabled
> (or
> > > as Staffan pointed out, running with a different tmp directory) then
> the
> > > target VM gets a SIGQUIT and just prints the thread dump. It's hard to
> say
> > > what Piotr is running into but I think this at least explains it for
> the
> > > minimal VM.
> > >
> > > -Alan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20131122/2a0b1eb6/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: strace-out.tgz
Type: application/x-gzip
Size: 3044 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20131122/2a0b1eb6/strace-out-0001.tgz