Questions about com.sun.tools.attach implementation on windows

[Thomas Stüfe]

Hi all,

I have a question about the implementation of com.sun.tools.attach on
Windows.

when investigating a crashes which were related to VirtualMachineImpl.c
(see https://bugs.openjdk.java.net/browse/JDK-8059868), the implementation
seemed to me very unusual and fragile:

We want to call monitoring functions in the target VM.

To do that, we inject a code stub (compiled by the c++ compiler used to
build the injector VM) into the target VM. Then we use CreateRemoteThread()
to run this code.

As a parameter to this function, we hand over addresses to Win32 APIS taken
from the injector-VM. The target VM will call those functions, implicitly
assuming the locations to be the same in both processes.

Problems which may occur (at least as far as I understand it):

- the code injected from the injector VM into the target VM must be
position independent - something which may easily break with new security
features Microsoft adds to their compiler (e.g. the /RTC1 crash reported
above).

- kernel32.dll must be always loaded at the same address for the Win32 API
calls to work. I am not sure this is always the case now, with address
layout randomization.

- I also could imagine (though I did not see that) virus scanners not being
happy and reporting this activity as suspicious.

Also, if the coding fails - most likely crashes - effects are a bit harsh:

- the target VM, which is "innocent" and my be stable and years old, will
crash if the injector VM was compiled wrong, e.g. in a way which makes the
injected code position dependend. This may happen by a new compiler switch
or compiler update.

- There is no way to get decent error handling, at least for 32bit, because
we rely on stack based SEH handling and in the remote thread there is no
SEH handler yet set up. The effect is sudden death
of the target VM with no hs-err file.

- The crash is difficult to figure out: first one will analyze the target
VM, which is innocent, because it crashes. But we have no matching debug
information for the injected stub, so debugging is difficult.

I would love to know why we do it this way. I am sure there is a valid
reason for it. Maybe Backward compatibility?

Thanks! and Kind Regards,

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20141009/c5c5e200/attachment.html>