<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Python C types [1] seems to go down a similar path:</p>
<p>
<blockquote type="cite">The <em>use_last_error</em> parameter,
when set to true, enables the same mechanism for
the Windows error code which is managed by the <a class="reference internal" href="https://docs.python.org/3/library/ctypes.html#ctypes.GetLastError" title="ctypes.GetLastError"><code class="xref py py-func
docutils literal notranslate"><span class="pre">GetLastError()</span></code></a>
and
<code class="xref py py-func docutils literal notranslate"><span class="pre">SetLastError()</span></code> Windows API
functions; <a class="reference internal" href="https://docs.python.org/3/library/ctypes.html#ctypes.get_last_error" title="ctypes.get_last_error"><code class="xref py py-func
docutils literal notranslate"><span class="pre">ctypes.get_last_error()</span></code></a>
and
<a class="reference internal" href="https://docs.python.org/3/library/ctypes.html#ctypes.set_last_error" title="ctypes.set_last_error"><code class="xref py py-func
docutils literal notranslate"><span class="pre">ctypes.set_last_error()</span></code></a>
are used to request and change the ctypes private
copy of the windows error code.</blockquote>
<br>
We currently have an enhancement open to add various kind of extra
linkage information [2]. I believe this is yet another case where
custom linkage information is required. I have updated that JBS
entry to capture some info in this thread.<br>
</p>
<p>Maurizio<br>
</p>
<p>[1] - <a class="moz-txt-link-freetext" href="https://docs.python.org/3/library/ctypes.html">https://docs.python.org/3/library/ctypes.html</a><br>
[2] - <a class="moz-txt-link-freetext" href="https://bugs.openjdk.org/browse/JDK-8292047">https://bugs.openjdk.org/browse/JDK-8292047</a><br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 16/08/2022 15:29, Manuel
Bleichenbacher wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAA7F5jJC59h8ZtB7b1s4Vf=wE2EQs5rNrQbgeHNm-K8T80BRZA@mail.gmail.com">
<div dir="ltr">
<div>Another option is to make it part of the function
descriptor / calling convention, similar to how .NET Interop
does it:</div>
<div><br>
</div>
<div>
<span style="color:rgb(23,23,23);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:pre;word-spacing:0px;background-color:rgb(242,242,242);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none"> [</span><span class="gmail-hljs-meta" style="box-sizing:inherit;outline-color:inherit;color:rgb(0,104,129);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:pre;word-spacing:0px;background-color:rgb(242,242,242);text-decoration-style:initial;text-decoration-color:initial">DllImportAttribute(<span class="gmail-hljs-string" style="box-sizing:inherit;outline-color:inherit;color:rgb(163,21,21)">"user32.dll"</span>, SetLastError = true, CharSet = CharSet.Unicode)</span><span style="color:rgb(23,23,23);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:pre;word-spacing:0px;background-color:rgb(242,242,242);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none">]
</span><span class="gmail-hljs-function" style="box-sizing:inherit;outline-color:inherit;color:rgb(23,23,23);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:pre;word-spacing:0px;background-color:rgb(242,242,242);text-decoration-style:initial;text-decoration-color:initial"><span class="gmail-hljs-keyword" style="box-sizing:inherit;outline-color:inherit;color:rgb(1,1,253)">public</span> <span class="gmail-hljs-keyword" style="box-sizing:inherit;outline-color:inherit;color:rgb(1,1,253)">static</span> <span class="gmail-hljs-keyword" style="box-sizing:inherit;outline-color:inherit;color:rgb(1,1,253)">extern</span> <span class="gmail-hljs-built_in" style="box-sizing:inherit;outline-color:inherit;color:rgb(1,1,253)">int</span> <span class="gmail-hljs-title" style="box-sizing:inherit;outline-color:inherit;color:rgb(0,104,129)">MessageBox</span>(<span class="gmail-hljs-params" style="box-sizing:inherit;outline-color:inherit">IntPtr hwnd, String text, String caption, uint type</span>)</span><span style="color:rgb(23,23,23);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:pre;word-spacing:0px;background-color:rgb(242,242,242);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none">;
</span></div>
<div><br>
</div>
<div>With the SetLastError annotation, this function call will
save the last error in a place where the .NET runtime
environment will not overwrite it. It can then be queried at
anytime using Marshal.GetLastWin32Error(), or rather at
anytime until calling the next Interop function. The method
call must be atomic of course.</div>
<div><br>
</div>
<div>A similar implementation would be needed for errno on
Linux.</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Aug 16, 2022 at 3:55
PM Thomas Stüfe <<a href="mailto:thomas.stuefe@gmail.com" moz-do-not-send="true" class="moz-txt-link-freetext">thomas.stuefe@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<div>A pragmatic and low-tech way to solve this could be to
build a library that wraps all (or a subset of) win32 APIs
and just reworks them slightly to return the error
(GetLastError or WSAGetLastError) as part of the call.
E.g. by prepending or appending a (DWORD* errorcode)
parameter. If we do this and ship this utility wrapper
library as part of the JDK, we can probably help 99% of
callers who have this problem.</div>
<div><br>
</div>
<div>..Thomas</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Aug 16, 2022 at
1:26 PM Maurizio Cimadamore <<a href="mailto:maurizio.cimadamore@oracle.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">maurizio.cimadamore@oracle.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>On 15/08/2022 21:49, Manuel Bleichenbacher wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I've run a test with jdb, and the
outcome is the same: the program behaves
incorrectly.
<div><br>
</div>
<div>Do you guys know if this is caused by the
debugger only, or if it is a general problem
that is more likely to occur with the debugger
but could also occur if run without debugger?</div>
</div>
</blockquote>
<p>Hi,<br>
as David pointed out, nothing in the JVM, debuggers,
agents etc. takes this use case into account - which
means that whether errno/LastError is preserved is,
ultimately, platform/OS-dependent.</p>
<p>In JNI the status quo has probably been good
enough: after all with JNI you can always insert
your errno/LastError check directly into the native
code, ensuring atomicity.</p>
<p>But when you use some kind of FFI, each Java call
corresponds to _single_ native call - which means
that, even if you want to do:</p>
<p><call function><br>
<getLastError></p>
<p>In reality there are some operations which "might"
happen between the two calls (after all, the JVM is
a complex piece of software, the garbage collector
might need to allocate more memory, etc.).</p>
<p>This explains why some of the frameworks out there
shadow (and save) errno/LastError into separate
thread-local storage, so that it can be read by
other FFI calls. What I've seen doesn't seem a
particularly compelling solution though, because:
(a) the FFI support has to add platform-dependent
logic to mimic what e.g. getLastError and
setLastError would do, and (b) the FFI support needs
to "rewire" calls to such native functions, so that
they return the shadowed storage instead. So you
could end up with cases where the native LastError
value and the FFI value are out of sync.<br>
</p>
<p>A more general solution would be to combine
multiple downcall method handles into one, so that
e.g. one could apply a GetLastError filter to a
downcall method handle, which calls GetLastError
after the native call, and saves the value into some
user-defined variable, and does that within the
_same_ native execution as the original call (e.g.
only one Java->native transition, covering both
calls). But the Linker API does not currently offer
such capability (and something like that would be a
bit on the boundary of what a method handle
combinator API is allowed to do).</p>
<p>Maurizio<br>
</p>
<p><br>
</p>
<blockquote type="cite"><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Aug 12,
2022 at 3:16 PM Maurizio Cimadamore <<a href="mailto:maurizio.cimadamore@oracle.com" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">maurizio.cimadamore@oracle.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Manuel,<br>
thanks for submitting this issue.</p>
<p>I think your hunch is probably correct -
something inside the JDK is resetting the
value of LastError.</p>
<p>In Hotspot code I see some abstractions to
preserve the LastError value
(os_windows.cpp):</p>
<pre style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:"Source Code Pro",monospace;font-size:11.3pt"><span style="color:rgb(128,128,128);font-style:italic">// A number of wrappers for more frequently used system calls, to add standard logging.
</span><span style="color:rgb(128,128,128);font-style:italic">
</span><span style="color:rgb(0,0,128);font-weight:bold">struct </span>PreserveLastError {
<span style="color:rgb(0,0,128);font-weight:bold">const </span>DWORD v;
PreserveLastError() : v(::GetLastError()) {}
~PreserveLastError() { ::SetLastError(v); }
};</pre>
<p><br>
And this is used in a number of OS-specific
function calls, to avoid polluting the last
error value.</p>
<p>That said, when you are running with a
debugger, especially inside an IDE (which
might add its own hooks), I think most bets
are off, as the debugger might indeed
perform additional native calls which might
not preserve lastError correctly.</p>
<p>One experiment would be to try and
debugging using jdb - so that at least we'd
rule the IDE out, and see if the issue is
still there. If that's the case we'll try to
reach out to somebody more intimate with
architetcure of JPDA, to see if that's
something that can be addressed (perhaps in
a way similar to what hotspot code seems to
be already doing).</p>
<p>Thanks<br>
Maurizio<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div>On 12/08/2022 10:21, Manuel
Bleichenbacher wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Thanks for the work on
project Panama. It's an exciting
technology. I'm using it to make native
operating system services available to
Java.
<div><br>
</div>
<div>On Windows I've run into an issue.
This C/C++ Windows code:
<div>
<div><br>
</div>
<div> BOOL res =
WriteFile(INVALID_HANDLE_VALUE,
NULL, 0, NULL, NULL);<br>
DWORD err = GetLastError();<br>
</div>
</div>
<div> printf("WriteFile result: %d,
GetLastError result: %d\n", res, err);<br>
</div>
<div><br>
</div>
<div>prints (as expected):</div>
<div><br>
</div>
<div> WriteFile result: 0,
GetLastError result: 6<br>
</div>
<div><br>
</div>
<div>6 is the value of the
constant ERROR_INVALID_HANDLE.</div>
<div><br>
</div>
<div>Using panama, I've translated the
code to Java. The result is:</div>
<div><br>
</div>
<div>Without debugger, the output is the
same. Everything is ok.</div>
<div><br>
</div>
<div>With the debugger, the result is
incorrect:</div>
<div><br>
</div>
<div>
<div> WriteFile result: 0,
GetLastError result: 0<br>
</div>
<div><br>
</div>
<div>This is incorrect. WriteFile()
indicates an error, but
GetLastError() returns 0
(= NO_ERROR). Could it be that the
debugger calls another Windows API
function between those two
functions, resetting the last error?</div>
<div><br>
</div>
<div>In my project that's a major
issue. Since the software behaves
incorrectly with the debugger, the
software can no longer be debugged.
This doesn't just affect it when
debugging this particular piece of
code but anytime this code is run in
a debugging session.</div>
<div><br>
</div>
<div>Is there something I'm not doing
incorrectly? Or is there a fix or
workaround?</div>
<div><br>
</div>
<div>Here's the Java code:</div>
<div><br>
</div>
<div>
<pre style="background-color:rgb(43,43,43);color:rgb(169,183,198);font-family:"JetBrains Mono",monospace;font-size:9.8pt"><span style="color:rgb(204,120,50)">import </span>java.lang.foreign.*<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)">import </span>java.lang.invoke.MethodHandle<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)">
</span><span style="color:rgb(204,120,50)">import static </span>java.lang.foreign.MemoryAddress.<span style="color:rgb(152,118,170);font-style:italic">NULL</span><span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)">import static </span>java.lang.foreign.ValueLayout.<span style="color:rgb(152,118,170);font-style:italic">ADDRESS</span><span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)">import static </span>java.lang.foreign.ValueLayout.<span style="color:rgb(152,118,170);font-style:italic">JAVA_INT</span><span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)">
</span><span style="color:rgb(204,120,50)">public class </span>WinApi {
<span style="color:rgb(204,120,50)">static final </span>MethodHandle <span style="color:rgb(152,118,170);font-style:italic">WriteFile$Func</span><span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> static final </span>MethodHandle <span style="color:rgb(152,118,170);font-style:italic">GetLastError$Func</span><span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> static final </span>MemoryAddress <span style="color:rgb(152,118,170);font-style:italic">INVALID_HANDLE_VALUE </span>= MemoryAddress.<span style="font-style:italic">ofLong</span>(-<span style="color:rgb(104,151,187)">1</span>)<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)">
</span><span style="color:rgb(204,120,50)"> static </span>{
<span style="color:rgb(204,120,50)">var </span>linker = Linker.<span style="font-style:italic">nativeLinker</span>()<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> var </span>lookup = SymbolLookup.<span style="font-style:italic">libraryLookup</span>(<span style="color:rgb(106,135,89)">"Kernel32"</span><span style="color:rgb(204,120,50)">, </span>MemorySession.<span style="font-style:italic">global</span>())<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)">
</span><span style="color:rgb(204,120,50)"> </span><span style="color:rgb(152,118,170);font-style:italic">WriteFile$Func </span>= linker.downcallHandle(
lookup.lookup(<span style="color:rgb(106,135,89)">"WriteFile"</span>).get()<span style="color:rgb(204,120,50)">,
</span><span style="color:rgb(204,120,50)"> </span>FunctionDescriptor.<span style="font-style:italic">of</span>(<span style="color:rgb(152,118,170);font-style:italic">JAVA_INT</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(152,118,170);font-style:italic">ADDRESS</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(152,118,170);font-style:italic">ADDRESS</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(152,118,170);font-style:italic">JAVA_INT</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(152,118,170);font-style:italic">ADDRESS</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(152,118,170);font-style:italic">ADDRESS</span>)
)<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> </span><span style="color:rgb(152,118,170);font-style:italic">GetLastError$Func </span>= linker.downcallHandle(
lookup.lookup(<span style="color:rgb(106,135,89)">"GetLastError"</span>).get()<span style="color:rgb(204,120,50)">,
</span><span style="color:rgb(204,120,50)"> </span>FunctionDescriptor.<span style="font-style:italic">of</span>(<span style="color:rgb(152,118,170);font-style:italic">JAVA_INT</span>)
)<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> </span>}
<span style="color:rgb(204,120,50)">public static void </span><span style="color:rgb(255,198,109)">main</span>(String[] args) {
<span style="color:rgb(204,120,50)">var </span>res = <span style="font-style:italic">WriteFile</span>(<span style="color:rgb(152,118,170);font-style:italic">INVALID_HANDLE_VALUE</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(152,118,170);font-style:italic">NULL</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(104,151,187)">0</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(152,118,170);font-style:italic">NULL</span><span style="color:rgb(204,120,50)">, </span><span style="color:rgb(152,118,170);font-style:italic">NULL</span>)<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> var </span>err = <span style="font-style:italic">GetLastError</span>()<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> </span>System.<span style="color:rgb(152,118,170);font-style:italic">out</span>.printf(<span style="color:rgb(106,135,89)">"WriteFile result: %d, GetLastError result: %d</span><span style="color:rgb(204,120,50)">\n</span><span style="color:rgb(106,135,89)">"</span><span style="color:rgb(204,120,50)">, </span>res<span style="color:rgb(204,120,50)">, </span>err)<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> </span>}
<span style="color:rgb(204,120,50)">static int </span><span style="color:rgb(255,198,109)">WriteFile</span>(MemoryAddress hFile<span style="color:rgb(204,120,50)">, </span>MemoryAddress lpBuffer<span style="color:rgb(204,120,50)">, int </span>nNumberOfBytesToWrite<span style="color:rgb(204,120,50)">,
</span><span style="color:rgb(204,120,50)"> </span>MemoryAddress lpNumberOfBytesWritten<span style="color:rgb(204,120,50)">, </span>MemoryAddress lpOverlapped) {
<span style="color:rgb(204,120,50)">try </span>{
<span style="color:rgb(204,120,50)">return </span>(<span style="color:rgb(204,120,50)">int</span>) <span style="color:rgb(152,118,170);font-style:italic">WriteFile$Func</span>.invokeExact((Addressable)hFile<span style="color:rgb(204,120,50)">, </span>(Addressable)lpBuffer<span style="color:rgb(204,120,50)">, </span>nNumberOfBytesToWrite<span style="color:rgb(204,120,50)">,
</span><span style="color:rgb(204,120,50)"> </span>(Addressable)lpNumberOfBytesWritten<span style="color:rgb(204,120,50)">, </span>(Addressable)lpOverlapped)<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> </span>} <span style="color:rgb(204,120,50)">catch </span>(Throwable e) {
<span style="color:rgb(204,120,50)">throw new </span>RuntimeException(e)<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> </span>}
}
<span style="color:rgb(204,120,50)">static int </span><span style="color:rgb(255,198,109)">GetLastError</span>() {
<span style="color:rgb(204,120,50)">try </span>{
<span style="color:rgb(204,120,50)">return </span>(<span style="color:rgb(204,120,50)">int</span>) <span style="color:rgb(152,118,170);font-style:italic">GetLastError$Func</span>.invokeExact()<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> </span>} <span style="color:rgb(204,120,50)">catch </span>(Throwable e) {
<span style="color:rgb(204,120,50)">throw new </span>RuntimeException(e)<span style="color:rgb(204,120,50)">;
</span><span style="color:rgb(204,120,50)"> </span>}
}
}</pre>
</div>
<div><br>
</div>
</div>
<div>Cheers</div>
<div>Manuel</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>