AttachCurrentThread Crash

[Markus KARG]

Hello Hotspot Community,

I experienced a problem when using JNI and since nobody else was able to 
help me since months, and since it looks like a bug in the jvm.dll, you 
are my last chance. :-)

In fact I am an experienced JNI user so my question is not about JNI 
itself but about a very complex case that makes jvm.dll crash down. So I 
want to discuss with you how we can solve it -- either by working around 
it in my code, or by adding a fix to the Hotspot VM (which seems to be 
the cause of the problems).

I am writing a DLL in Windows XP that itself is a plugin to the desktop, 
a.k.a "Shell Extension". The DLL itself runs really stable (in fact I 
have drilled down so far that it actually does nothing anymore now, so I 
am pretty sure there is no bug in it since there is no more "real" code 
in it besides creating the Java VM). explorer.exe (i. e. the desktop 
process of Windows) loads the DLL, executes my code, and unloads the DLL 
then. So far, so good, works pretty well.

But the same Windows process then loads my DLL once more, and executes 
it again (just as it did before). My DLL, you can imagine, is using JNI 
to call some Java code. Certainly I was clever enough not to create a VM 
twice, since I certainly know that both is impossible, either creating 
two VMs in the same process, or to try to create a VM after the previous 
VM was deleted (which I think is a bug, but anyways, that is not the 
point here).

So I am checking first wheter there is a VM already using 
JNI_GetCreatedJavaVMs which returns the number of 1 and a pointer to the 
JavaVM. Great. That works pretty well. But if I now try to call 
AttachCurrentThread to get a pointer to the Env, then the desktop 
totally crashs down, telling me that I tried to execute code that 
actually is data (I switched off Data Execution Prevention then, but now 
it crashs down without ANY further note, so it is not a real help). So 
why did that happen?

I looked at the source code of the JVM and it seems as if it uses global 
variables very heavily in the JNI source files. I could imagine that in 
fact one of those pointers now (after explorer unloaded and reloaded my 
DLL in the same process) point to a memory page that is marked as "data" 
but not as "code", so when using the function pointer, windows crashs 
down (since the memory pointed to is presumably no more part of the 
process's memory map).

Please help me. I do not have any idea how to fix that. As a workaround 
I told Windows not to ever unload my DLL (what is working), but that 
certainly is just a bad hack. The real solution must be to convince 
jvm.dll to attach the thread without crashing down.

If needed, I can provide an "at most simple" project that proofs the 
behaviour on any Windows machine (without seriously damaging it).

Thanks!
Markus

-- 
http://www.xing.com/go/invita/58469

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3505 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20070808/7c71c953/attachment.bin