jstack and the target output

[Mikael Gerdin]

Piotr,

On Friday 22 November 2013 09.21.56 Piotr Bzdyl wrote:
> Is there any other information I could collect to help troubleshoot this
> problem? How could I check if VM runs with attach disabled?

The comments in 
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java
describe the handshaking involved in dynamically enabling the attach mechanism 
on Linux.

The mechanism is basically:
* The attacher checks for the existence of the socket file 
/tmp/.java_pid${PID}
* If that does not exist it creates a file .attach_pid${PID} in the target 
process' working directory and sends a SIGQUIT (kill -QUIT) to that process.
* The SIGQUIT handler notices the .attach_pid${PID} file and creates the 
socket at /tmp/.java_pid${PID}
* The attacher notices the socket file and unlinks the .attach_pid${PID} file.

You can force the target process to always create the socket file at startup 
by using the -XX:+StartAttachListener command line flag.

To further troubleshoot the problem I suggest that you try strace:ing the 
target process and jstack with the following flags enabled:

(target process) 
$ strace -f -e trace=accept,stat,socket,bind $JAVA_HOME/bin/java donothing

(jstack process)
$ strace -f -e trace=open,connect,stat,unlink,write,kill $JAVA_HOME/bin/jstack 
<pid>

You should get something similar to the following:
(target process)
(...)
[pid 12602] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=12687, 
si_uid=726} ---
[pid 12630] stat(".attach_pid12602", {st_mode=S_IFREG|0644, st_size=0, ...}) = 
0
Process 12733 attached
[pid 12733] socket(PF_LOCAL, SOCK_STREAM, 0) = 5
[pid 12733] bind(5, {sa_family=AF_LOCAL, sun_path="/tmp/.java_pid12602.tmp"}, 
110) = 0
[pid 12733] accept(5, {sa_family=AF_LOCAL, NULL}, [2]) = 6

(jstack process)
[pid 12688] stat("/localhome/java/jdk-8-ea-bin-
b112/jre/lib/amd64/libattach.so", {st_mode=S_IFREG|0755, st_size=16029, ...}) 
= 0
[pid 12688] open("/localhome/java/jdk-8-ea-bin-
b112/jre/lib/amd64/libattach.so", O_RDONLY) = 7
[pid 12688] open("/localhome/java/jdk-8-ea-bin-
b112/jre/lib/amd64/libattach.so", O_RDONLY|O_CLOEXEC) = 7
[pid 12688] stat("/tmp/.java_pid12602", 0x7fe53183e3a0) = -1 ENOENT (No such 
file or directory)
[pid 12688] open("/proc/12602/cwd/.attach_pid12602", O_RDWR|O_CREAT|O_EXCL, 
0666) = 7
[pid 12688] kill(12602, SIGQUIT)        = 0
[pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, st_size=0, 
...}) = 0
[pid 12688] unlink("/proc/12602/cwd/.attach_pid12602") = 0
[pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600, st_size=0, 
...}) = 0
[pid 12688] connect(7, {sa_family=AF_LOCAL, sun_path="/tmp/.java_pid12602"}, 
110) = 0
[pid 12688] connect(7, {sa_family=AF_LOCAL, sun_path="/tmp/.java_pid12602"}, 
110) = 0
[pid 12688] write(7, "1", 1)            = 1
[pid 12688] write(7, "\0", 1)           = 1
[pid 12688] write(7, "threaddump", 10)  = 10
[pid 12688] write(7, "\0", 1)           = 1

/Mikael

> 
> I noticed the same behavior when trying to connect to the process using
> JConsole: JConsole displays the sample app in the list and when I try to
> connect it finally displays an error that it cannot connect whereas the
> sample app prints thread dump on its console.
> 
> On Thu, Nov 21, 2013 at 7:18 PM, Alan Bateman 
<Alan.Bateman at oracle.com>wrote:
> > On 21/11/2013 14:47, David Holmes wrote:
> >> I can recreate this using the minimal VM, which doesn't support
> >> libattach. So it definitely seems like something is going wrong in the
> >> attach process, but I can't say what. Not sure what debugging hooks we
> >> have
> >> for the attach mechanism ??
> > 
> > The first attach involves signaling the target VM to start up the attach
> > mechanism. So for the minimal VM or a VM running with attach disabled (or
> > as Staffan pointed out, running with a different tmp directory) then the
> > target VM gets a SIGQUIT and just prints the thread dump. It's hard to say
> > what Piotr is running into but I think this at least explains it for the
> > minimal VM.
> > 
> > -Alan