GetLastError() (with and without debugger)

[Manuel Bleichenbacher]

Thanks for the work on project Panama. It's an exciting technology. I'm
using it to make native operating system services available to Java.

On Windows I've run into an issue. This C/C++ Windows code:

    BOOL res = WriteFile(INVALID_HANDLE_VALUE, NULL, 0, NULL, NULL);
    DWORD err = GetLastError();
    printf("WriteFile result: %d, GetLastError result: %d\n", res, err);

prints (as expected):

    WriteFile result: 0, GetLastError result: 6

6 is the value of the constant ERROR_INVALID_HANDLE.

Using panama, I've translated the code to Java. The result is:

Without debugger, the output is the same. Everything is ok.

With the debugger, the result is incorrect:

    WriteFile result: 0, GetLastError result: 0

This is incorrect. WriteFile() indicates an error, but GetLastError()
returns 0 (= NO_ERROR). Could it be that the debugger calls another Windows
API function between those two functions, resetting the last error?

In my project that's a major issue. Since the software behaves incorrectly
with the debugger, the software can no longer be debugged. This doesn't
just affect it when debugging this particular piece of code but anytime
this code is run in a debugging session.

Is there something I'm not doing incorrectly? Or is there a fix or
workaround?

Here's the Java code:

import java.lang.foreign.*;
import java.lang.invoke.MethodHandle;

import static java.lang.foreign.MemoryAddress.NULL;
import static java.lang.foreign.ValueLayout.ADDRESS;
import static java.lang.foreign.ValueLayout.JAVA_INT;

public class WinApi {
    static final MethodHandle WriteFile$Func;
    static final MethodHandle GetLastError$Func;
    static final MemoryAddress INVALID_HANDLE_VALUE = MemoryAddress.ofLong(-1);

    static {
        var linker = Linker.nativeLinker();
        var lookup = SymbolLookup.libraryLookup("Kernel32",
MemorySession.global());

        WriteFile$Func = linker.downcallHandle(
                lookup.lookup("WriteFile").get(),
                FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS,
JAVA_INT, ADDRESS, ADDRESS)
        );
        GetLastError$Func = linker.downcallHandle(
                lookup.lookup("GetLastError").get(),
                FunctionDescriptor.of(JAVA_INT)
        );
    }

    public static void main(String[] args) {
        var res = WriteFile(INVALID_HANDLE_VALUE, NULL, 0, NULL, NULL);
        var err = GetLastError();
        System.out.printf("WriteFile result: %d, GetLastError result:
%d\n", res, err);
    }

    static int WriteFile(MemoryAddress hFile, MemoryAddress lpBuffer,
int nNumberOfBytesToWrite,
                         MemoryAddress lpNumberOfBytesWritten,
MemoryAddress lpOverlapped) {
        try {
            return (int)
WriteFile$Func.invokeExact((Addressable)hFile, (Addressable)lpBuffer,
nNumberOfBytesToWrite,
                    (Addressable)lpNumberOfBytesWritten,
(Addressable)lpOverlapped);
        } catch (Throwable e) {
            throw new RuntimeException(e);
        }
    }

    static int GetLastError() {
        try {
            return (int) GetLastError$Func.invokeExact();
        } catch (Throwable e) {
            throw new RuntimeException(e);
        }
    }
}


Cheers
Manuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/panama-dev/attachments/20220812/94659944/attachment.htm>