jstack and the target output

Fri Nov 22 04:17:28 PST 2013

On Friday 22 November 2013 12.59.42 Piotr Bzdyl wrote:
> After checking the strace output I notice that the issue was caused by the
> file mode for files created on vboxsf mount. By default (when automount
> option is used in virtualbox configuration) it creates .attach_pid<pid>
> with file mode 0770. When I manually mounted shared folder specifying
> fmode=660 jstack worked as expected.
> 
> Thank you for pointing me to the right path to narrow down the issue.
> 
> BTW, where I could find the JVM code responsible for handling SIGQUIT
> signal to see what it does that file mode matters?

http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/os/linux/vm/attachListener_linux.cpp

and

http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/share/vm/runtime/os.cpp
(see signal_thread_entry and its call to AttachListener::is_init_trigger()

/Mikael


> 
> On Fri, Nov 22, 2013 at 10:30 AM, Piotr Bzdyl <piotr at bzdyl.net> wrote:
> > Thank you Mikael for the detailed information - I learned a lot about how
> > SA tools attache to the target process.
> > 
> > When I was running the test you suggested I noticed one "detail" in my
> > setup which turned out to be the cause of my problems: as I mentioned
> > previously I was running my tests inside a virtual machine (namely
> > VirtualBox). The issue was that the cwd dir of my target process was on
> > the
> > mounted shared directory (vboxsf filesystem) in VirtualBox VM and it was
> > causing the problems. I still have to go through the strace outputs to
> > find
> > out if it is because of the access rights of because how vboxsf file
> > system
> > driver works (attached the output).
> > 
> > On Fri, Nov 22, 2013 at 9:51 AM, Mikael Gerdin 
<mikael.gerdin at oracle.com>wrote:
> >> Piotr,
> >> 
> >> On Friday 22 November 2013 09.21.56 Piotr Bzdyl wrote:
> >> > Is there any other information I could collect to help troubleshoot
> >> > this
> >> > problem? How could I check if VM runs with attach disabled?
> >> 
> >> The comments in
> >> 
> >> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/solaris/classes/sun
> >> /tools/attach/LinuxVirtualMachine.java describe the handshaking involved
> >> in dynamically enabling the attach mechanism
> >> on Linux.
> >> 
> >> The mechanism is basically:
> >> * The attacher checks for the existence of the socket file
> >> /tmp/.java_pid${PID}
> >> * If that does not exist it creates a file .attach_pid${PID} in the
> >> target
> >> process' working directory and sends a SIGQUIT (kill -QUIT) to that
> >> process.
> >> * The SIGQUIT handler notices the .attach_pid${PID} file and creates the
> >> socket at /tmp/.java_pid${PID}
> >> * The attacher notices the socket file and unlinks the .attach_pid${PID}
> >> file.
> >> 
> >> You can force the target process to always create the socket file at
> >> startup
> >> by using the -XX:+StartAttachListener command line flag.
> >> 
> >> To further troubleshoot the problem I suggest that you try strace:ing the
> >> target process and jstack with the following flags enabled:
> >> 
> >> (target process)
> >> $ strace -f -e trace=accept,stat,socket,bind $JAVA_HOME/bin/java
> >> donothing
> >> 
> >> (jstack process)
> >> $ strace -f -e trace=open,connect,stat,unlink,write,kill
> >> $JAVA_HOME/bin/jstack
> >> <pid>
> >> 
> >> You should get something similar to the following:
> >> (target process)
> >> (...)
> >> [pid 12602] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=12687,
> >> si_uid=726} ---
> >> [pid 12630] stat(".attach_pid12602", {st_mode=S_IFREG|0644, st_size=0,
> >> ...}) =
> >> 0
> >> Process 12733 attached
> >> [pid 12733] socket(PF_LOCAL, SOCK_STREAM, 0) = 5
> >> [pid 12733] bind(5, {sa_family=AF_LOCAL,
> >> sun_path="/tmp/.java_pid12602.tmp"},
> >> 110) = 0
> >> [pid 12733] accept(5, {sa_family=AF_LOCAL, NULL}, [2]) = 6
> >> 
> >> (jstack process)
> >> [pid 12688] stat("/localhome/java/jdk-8-ea-bin-
> >> b112/jre/lib/amd64/libattach.so", {st_mode=S_IFREG|0755, st_size=16029,
> >> ...})
> >> = 0
> >> [pid 12688] open("/localhome/java/jdk-8-ea-bin-
> >> b112/jre/lib/amd64/libattach.so", O_RDONLY) = 7
> >> [pid 12688] open("/localhome/java/jdk-8-ea-bin-
> >> b112/jre/lib/amd64/libattach.so", O_RDONLY|O_CLOEXEC) = 7
> >> [pid 12688] stat("/tmp/.java_pid12602", 0x7fe53183e3a0) = -1 ENOENT (No
> >> such
> >> file or directory)
> >> [pid 12688] open("/proc/12602/cwd/.attach_pid12602",
> >> O_RDWR|O_CREAT|O_EXCL,
> >> 0666) = 7
> >> [pid 12688] kill(12602, SIGQUIT)        = 0
> >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600,
> >> st_size=0,
> >> ...}) = 0
> >> [pid 12688] unlink("/proc/12602/cwd/.attach_pid12602") = 0
> >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600,
> >> st_size=0,
> >> ...}) = 0
> >> [pid 12688] connect(7, {sa_family=AF_LOCAL,
> >> sun_path="/tmp/.java_pid12602"},
> >> 110) = 0
> >> [pid 12688] connect(7, {sa_family=AF_LOCAL,
> >> sun_path="/tmp/.java_pid12602"},
> >> 110) = 0
> >> [pid 12688] write(7, "1", 1)            = 1
> >> [pid 12688] write(7, "\0", 1)           = 1
> >> [pid 12688] write(7, "threaddump", 10)  = 10
> >> [pid 12688] write(7, "\0", 1)           = 1
> >> 
> >> /Mikael
> >> 
> >> > I noticed the same behavior when trying to connect to the process using
> >> > JConsole: JConsole displays the sample app in the list and when I try
> >> > to
> >> > connect it finally displays an error that it cannot connect whereas the
> >> > sample app prints thread dump on its console.
> >> > 
> >> > On Thu, Nov 21, 2013 at 7:18 PM, Alan Bateman
> >> 
> >> <Alan.Bateman at oracle.com>wrote:
> >> > > On 21/11/2013 14:47, David Holmes wrote:
> >> > >> I can recreate this using the minimal VM, which doesn't support
> >> > >> libattach. So it definitely seems like something is going wrong in
> >> 
> >> the
> >> 
> >> > >> attach process, but I can't say what. Not sure what debugging hooks
> >> 
> >> we
> >> 
> >> > >> have
> >> > >> for the attach mechanism ??
> >> > > 
> >> > > The first attach involves signaling the target VM to start up the
> >> 
> >> attach
> >> 
> >> > > mechanism. So for the minimal VM or a VM running with attach disabled
> >> 
> >> (or
> >> 
> >> > > as Staffan pointed out, running with a different tmp directory) then
> >> 
> >> the
> >> 
> >> > > target VM gets a SIGQUIT and just prints the thread dump. It's hard
> >> 
> >> to say
> >> 
> >> > > what Piotr is running into but I think this at least explains it for
> >> 
> >> the
> >> 
> >> > > minimal VM.
> >> > > 
> >> > > -Alan