jstack and the target output

Fri Nov 22 05:01:53 PST 2013

Thank you Mikael.

Now with the is_init_trigger() code I understand that it wasn't the problem
with file mask but rather with the file owner (lines 504-507 from
http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/os/linux/vm/attachListener_linux.cpp
).

What helped was specyfing uid i gid options to mount where provided IDs
were IDs of the user running jvm.

On Fri, Nov 22, 2013 at 1:17 PM, Mikael Gerdin <mikael.gerdin at oracle.com>wrote:

> On Friday 22 November 2013 12.59.42 Piotr Bzdyl wrote:
> > After checking the strace output I notice that the issue was caused by
> the
> > file mode for files created on vboxsf mount. By default (when automount
> > option is used in virtualbox configuration) it creates .attach_pid<pid>
> > with file mode 0770. When I manually mounted shared folder specifying
> > fmode=660 jstack worked as expected.
> >
> > Thank you for pointing me to the right path to narrow down the issue.
> >
> > BTW, where I could find the JVM code responsible for handling SIGQUIT
> > signal to see what it does that file mode matters?
>
>
> http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/os/linux/vm/attachListener_linux.cpp
>
> and
>
>
> http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/file/tip/src/share/vm/runtime/os.cpp
> (see signal_thread_entry and its call to AttachListener::is_init_trigger()
>
> /Mikael
>
>
> >
> > On Fri, Nov 22, 2013 at 10:30 AM, Piotr Bzdyl <piotr at bzdyl.net> wrote:
> > > Thank you Mikael for the detailed information - I learned a lot about
> how
> > > SA tools attache to the target process.
> > >
> > > When I was running the test you suggested I noticed one "detail" in my
> > > setup which turned out to be the cause of my problems: as I mentioned
> > > previously I was running my tests inside a virtual machine (namely
> > > VirtualBox). The issue was that the cwd dir of my target process was on
> > > the
> > > mounted shared directory (vboxsf filesystem) in VirtualBox VM and it
> was
> > > causing the problems. I still have to go through the strace outputs to
> > > find
> > > out if it is because of the access rights of because how vboxsf file
> > > system
> > > driver works (attached the output).
> > >
> > > On Fri, Nov 22, 2013 at 9:51 AM, Mikael Gerdin
> <mikael.gerdin at oracle.com>wrote:
> > >> Piotr,
> > >>
> > >> On Friday 22 November 2013 09.21.56 Piotr Bzdyl wrote:
> > >> > Is there any other information I could collect to help troubleshoot
> > >> > this
> > >> > problem? How could I check if VM runs with attach disabled?
> > >>
> > >> The comments in
> > >>
> > >>
> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/solaris/classes/sun
> > >> /tools/attach/LinuxVirtualMachine.java describe the handshaking
> involved
> > >> in dynamically enabling the attach mechanism
> > >> on Linux.
> > >>
> > >> The mechanism is basically:
> > >> * The attacher checks for the existence of the socket file
> > >> /tmp/.java_pid${PID}
> > >> * If that does not exist it creates a file .attach_pid${PID} in the
> > >> target
> > >> process' working directory and sends a SIGQUIT (kill -QUIT) to that
> > >> process.
> > >> * The SIGQUIT handler notices the .attach_pid${PID} file and creates
> the
> > >> socket at /tmp/.java_pid${PID}
> > >> * The attacher notices the socket file and unlinks the
> .attach_pid${PID}
> > >> file.
> > >>
> > >> You can force the target process to always create the socket file at
> > >> startup
> > >> by using the -XX:+StartAttachListener command line flag.
> > >>
> > >> To further troubleshoot the problem I suggest that you try strace:ing
> the
> > >> target process and jstack with the following flags enabled:
> > >>
> > >> (target process)
> > >> $ strace -f -e trace=accept,stat,socket,bind $JAVA_HOME/bin/java
> > >> donothing
> > >>
> > >> (jstack process)
> > >> $ strace -f -e trace=open,connect,stat,unlink,write,kill
> > >> $JAVA_HOME/bin/jstack
> > >> <pid>
> > >>
> > >> You should get something similar to the following:
> > >> (target process)
> > >> (...)
> > >> [pid 12602] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER,
> si_pid=12687,
> > >> si_uid=726} ---
> > >> [pid 12630] stat(".attach_pid12602", {st_mode=S_IFREG|0644, st_size=0,
> > >> ...}) =
> > >> 0
> > >> Process 12733 attached
> > >> [pid 12733] socket(PF_LOCAL, SOCK_STREAM, 0) = 5
> > >> [pid 12733] bind(5, {sa_family=AF_LOCAL,
> > >> sun_path="/tmp/.java_pid12602.tmp"},
> > >> 110) = 0
> > >> [pid 12733] accept(5, {sa_family=AF_LOCAL, NULL}, [2]) = 6
> > >>
> > >> (jstack process)
> > >> [pid 12688] stat("/localhome/java/jdk-8-ea-bin-
> > >> b112/jre/lib/amd64/libattach.so", {st_mode=S_IFREG|0755,
> st_size=16029,
> > >> ...})
> > >> = 0
> > >> [pid 12688] open("/localhome/java/jdk-8-ea-bin-
> > >> b112/jre/lib/amd64/libattach.so", O_RDONLY) = 7
> > >> [pid 12688] open("/localhome/java/jdk-8-ea-bin-
> > >> b112/jre/lib/amd64/libattach.so", O_RDONLY|O_CLOEXEC) = 7
> > >> [pid 12688] stat("/tmp/.java_pid12602", 0x7fe53183e3a0) = -1 ENOENT
> (No
> > >> such
> > >> file or directory)
> > >> [pid 12688] open("/proc/12602/cwd/.attach_pid12602",
> > >> O_RDWR|O_CREAT|O_EXCL,
> > >> 0666) = 7
> > >> [pid 12688] kill(12602, SIGQUIT)        = 0
> > >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600,
> > >> st_size=0,
> > >> ...}) = 0
> > >> [pid 12688] unlink("/proc/12602/cwd/.attach_pid12602") = 0
> > >> [pid 12688] stat("/tmp/.java_pid12602", {st_mode=S_IFSOCK|0600,
> > >> st_size=0,
> > >> ...}) = 0
> > >> [pid 12688] connect(7, {sa_family=AF_LOCAL,
> > >> sun_path="/tmp/.java_pid12602"},
> > >> 110) = 0
> > >> [pid 12688] connect(7, {sa_family=AF_LOCAL,
> > >> sun_path="/tmp/.java_pid12602"},
> > >> 110) = 0
> > >> [pid 12688] write(7, "1", 1)            = 1
> > >> [pid 12688] write(7, "\0", 1)           = 1
> > >> [pid 12688] write(7, "threaddump", 10)  = 10
> > >> [pid 12688] write(7, "\0", 1)           = 1
> > >>
> > >> /Mikael
> > >>
> > >> > I noticed the same behavior when trying to connect to the process
> using
> > >> > JConsole: JConsole displays the sample app in the list and when I
> try
> > >> > to
> > >> > connect it finally displays an error that it cannot connect whereas
> the
> > >> > sample app prints thread dump on its console.
> > >> >
> > >> > On Thu, Nov 21, 2013 at 7:18 PM, Alan Bateman
> > >>
> > >> <Alan.Bateman at oracle.com>wrote:
> > >> > > On 21/11/2013 14:47, David Holmes wrote:
> > >> > >> I can recreate this using the minimal VM, which doesn't support
> > >> > >> libattach. So it definitely seems like something is going wrong
> in
> > >>
> > >> the
> > >>
> > >> > >> attach process, but I can't say what. Not sure what debugging
> hooks
> > >>
> > >> we
> > >>
> > >> > >> have
> > >> > >> for the attach mechanism ??
> > >> > >
> > >> > > The first attach involves signaling the target VM to start up the
> > >>
> > >> attach
> > >>
> > >> > > mechanism. So for the minimal VM or a VM running with attach
> disabled
> > >>
> > >> (or
> > >>
> > >> > > as Staffan pointed out, running with a different tmp directory)
> then
> > >>
> > >> the
> > >>
> > >> > > target VM gets a SIGQUIT and just prints the thread dump. It's
> hard
> > >>
> > >> to say
> > >>
> > >> > > what Piotr is running into but I think this at least explains it
> for
> > >>
> > >> the
> > >>
> > >> > > minimal VM.
> > >> > >
> > >> > > -Alan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20131122/25afbfe1/attachment-0001.html 