GetLastError() (with and without debugger)

[Maurizio Cimadamore]

On 12/08/2022 15:55, Pedro Lamarão wrote:
> Em sex., 12 de ago. de 2022 às 11:50, Maurizio Cimadamore 
> <maurizio.cimadamore at oracle.com> escreveu:
>
>
>     On 12/08/2022 15:43, Pedro Lamarão wrote:
>>
>>         This is incorrect. WriteFile() indicates an error, but
>>         GetLastError() returns 0 (= NO_ERROR). Could it be that the
>>         debugger calls another Windows API function between those two
>>         functions, resetting the last error?
>>
>>         In my project that's a major issue. Since the software
>>         behaves incorrectly with the debugger, the software can no
>>         longer be debugged. This doesn't just affect it when
>>         debugging this particular piece of code but anytime this code
>>         is run in a debugging session.
>>
>>
>>
>>     This would be a blocker for all applications of Windows API
>>     functions via panama.
>>     Windows Sockets has a similar mechanism called WSAGetLastError.
>
>     And JNI - I doubt there's anything Panama specific here (unless
>     proven otherwise).
>
>
> Of course, this would disturb any interaction with Windows API, 
> however it was activated.
> For some reason, such a debugger has been working with software for 
> some time, and I presume it works with software that calls the system 
> via JNI.
> Is there perhaps some kind of special "locking" when calling into JNI 
> which the debugger might have learned to respect?

I think what we need at this point is some more testing.

E.g. can we reproduce the same issue with JNI? E.g. assume we have a JNI 
binding for Windows API we want to call and a _different_ JNI binding 
for checking LastError. Does that show the same problem?

My gut feeling is that when writing this in JNI, developers would have 
rolled in the LastError code check directly inside the native code. And 
the native code execution is opaque to debuggers, so I don't think a 
debugger could alter the LastError value seen by the JNI native code.

But in this case there are _two_ calls, one to "WriteFile" and one to 
"GetLastError". So the debugger would have plenty of opportunities to 
make other system calls in between.

As I said, I believe this should be visible in JNI too - but perhaps 
nobody would write code like this using JNI. E.g. with JNI, since it's 
painful enough to write one native method, developers tend to "lump" 
multiple functionalities, e.g. error checking, in the same native method 
implementation. Which, coincidentally, bypasses some issues that might 
be visible when using debuggers (so it is possible that this is a 
problem that debuggers never had to worry too much about).

Maurizio

>
> -- 
> Pedro Lamarão
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20220812/ef6c8ac1/attachment.htm>