RFR: 8374469: Mixed jstack does not work on Windows [v2]

[Yasumasa Suenaga]

On Thu, 15 Jan 2026 04:52:19 GMT, Chris Plummer <cjplummer at openjdk.org> wrote:

>> I don't know why these call stacks were produced... It's strange.
>> I tested this PR with cl.exe 19.50.35721 by VS 2026 18.1.1, I didn't see such call stacks.
>> 
>> `callJNI()` makes recursive call, but it should include Java upcall (you can see it as LingeredAppWithNativeMethod.callback()`)
>> 
>> Top of call stacks is `callJNI()`, thus I believe it has valid RSP/RBP/RIP (passed by `WindbgCDebugger::topFrameForThread`). `callJNI` is JNI function in NoFramePointer.dll generated by cl.exe (in usual (I believe) on OpenJDK build environment), thus I think `GetStackTrace` API works.
>> Each `NoFramePointer!Java_LingeredAppWithNativeMethod_callJNI` has instruction address, it means they were unwinded by the API, thus I think we need to investigate assembly code in NoFramePointer.dll to know what happens...
>
> The binary I have was built with VS2022-17.13.2.

I saw same lines on [GHA results](https://github.com/YaSuenag/jdk/actions/runs/20984170537/artifacts/5123260970) - it used VS 2022 to build. Repeats of `NoFramePointer!Java_LingeredAppWithNativeMethod_callJNI` is expected. It is inlined `fib()` as you said.

I saw callJNI + 0xdb and 0xd0 are repeated as following:


0x00007ffa9b0e10db	NoFramePointer!Java_LingeredAppWithNativeMethod_callJNI + 0xdb
0x00007ffa9b0e10d0	NoFramePointer!Java_LingeredAppWithNativeMethod_callJNI + 0xd0
0x00007ffa9b0e10db	NoFramePointer!Java_LingeredAppWithNativeMethod_callJNI + 0xdb
0x00007ffa9b0e10d0	NoFramePointer!Java_LingeredAppWithNativeMethod_callJNI + 0xd0


I got [GHA artifacts of debug build](https://github.com/YaSuenag/jdk/actions/runs/20984170537/artifacts/5122848054) and disassembled NoFramePointer.dll, then I saw following code around callJNI + 0xdb and 0xd0:

  00000001800010A8: 48 89 5C 24 08     mov         qword ptr [rsp+8],rbx
  00000001800010AD: 57                 push        rdi
  00000001800010AE: 48 83 EC 20        sub         rsp,20h
  00000001800010B2: 8B F9              mov         edi,ecx
  00000001800010B4: 85 C9              test        ecx,ecx
  00000001800010B6: 75 04              jne         00000001800010BC
  00000001800010B8: 33 C0              xor         eax,eax
  00000001800010BA: EB 22              jmp         00000001800010DE
  00000001800010BC: 83 FF 02           cmp         edi,2
  00000001800010BF: 7F 07              jg          00000001800010C8
  00000001800010C1: B8 01 00 00 00     mov         eax,1
  00000001800010C6: EB 16              jmp         00000001800010DE
  00000001800010C8: 83 C1 FE           add         ecx,0FFFFFFFEh
  00000001800010CB: E8 D8 FF FF FF     call        00000001800010A8
  00000001800010D0: 8D 4F FF           lea         ecx,[rdi-1]
  00000001800010D3: 48 8B D8           mov         rbx,rax
  00000001800010D6: E8 CD FF FF FF     call        00000001800010A8
  00000001800010DB: 48 03 C3           add         rax,rbx


It makes recursive call to 0x1800010A8 (callJNI + 0xa8) until the condition is met. According to NoFramePointer.dll.map in build artifact, callJNI + 0xa8 is `fib()` in libNoFramePointer.c.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/29019#discussion_r2693380875