RFR: 8293862: javax/swing/JFileChooser/8046391/bug8046391.java failed with 'Cannot invoke "java.awt.Image.getWidth(java.awt.image.ImageObserver)" because "retVal" is null'

[Karl T]

On Mon, 16 Jan 2023 14:19:52 GMT, Alexey Ivanov <aivanov at openjdk.org> wrote:

> This is somewhat a continuation of #11104 where the issue was discussed and where I found [the root cause](https://github.com/openjdk/jdk/pull/11104#issuecomment-1382435784).
> 
> **Root Cause**
> 
> The icon extraction code compares the returned handle to zero: 
> https://github.com/openjdk/jdk/blob/8cb25d3de494c6d9357a1c199e1a9dbff9be9948/src/java.desktop/windows/classes/sun/awt/shell/Win32ShellFolder2.java#L1157-L1158
> 
> On 64-bit systems, a valid handle can be negative. For example:
> 
> 
> getIcon : Desktop 16(16)
> GetIconLocation(uFlags=0x22, flags=0x4, index=-110) SUCCESS - szBuf=C:\Windows\system32\imageres.dll
> Extract - hres: 0, hIcon=0000000019B200F9, hIconSmall=FFFFFFFFF98C00EB, size=16(0x100020)
> SUCCEEDED
> !!! hIcon(0xfffffffff98c00eb) <= 0 : Desktop 16(16)
> 
> 
> Here in `size=16(0x100020)`, 16 is the requested icon size, the value in parenthesis is value of `iconSize` passed to the [`Extract`](https://learn.microsoft.com/en-us/windows/win32/api/shlobj_core/nf-shlobj_core-iextracticonw-extract) method that contains the size for small icon in high word and that for large icon in low word. `Desktop 16(16)` means the icon of size 16×16 is requested, and we're extracting 16×16 icon.
> 
> The icon was extracted _successfully_ but its handle was interpreted as an error.
> 
> The same problem exists in the fallback code:
> https://github.com/openjdk/jdk/blob/8cb25d3de494c6d9357a1c199e1a9dbff9be9948/src/java.desktop/windows/classes/sun/awt/shell/Win32ShellFolder2.java#L1161
> 
> Then when the fallback code is executed, another problem occurs:
> 
> 
> GetIconLocation(uFlags=0x40, flags=0x1a3, index=-675088752) SUCCESS - szBuf=
> Extract - hres: 80004005, hIcon=0000000000000000, hIconSmall=000001A3D7C2F83A, size=16(100020)
> NOT SUCCEEDED
> hIcon = 0x1a3d7c2f83a : Desktop 16(16)
> 
> 
> The error code 0x80004005 is `E_FAIL`. Yet the return value is not verified in the code:
> https://github.com/openjdk/jdk/blob/8cb25d3de494c6d9357a1c199e1a9dbff9be9948/src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp#L997-L1003
> 
> Since the icon handle is *seemingly* valid, it continues to extracting the icon bits:
> 
> 
> native getIconBits - fn_GetIconInfo returned false
> Error code: 0x57a - Invalid cursor handle.
> makeIcon: iconBits = null
> 
> 
> It's not a surprise. After all, the `Extract` method returned an error, which means neither icon handle is valid.
> 
> In the end, the `MultiResolutionIcon` contains `null`:
> 
>     16 -> null
>     32 -> BufferedImage @ 59e735d
>     24 -> BufferedImage @ 60b2faf0
> 
> 
> This problem has existed fo...

This PR does **not fix** the NPE. It is **still there in JDK 21.0.1**.

See [this post](https://forums.oracle.com/ords/apexds/post/jdk-21-0-1-the-open-file-dialog-hangs-when-selecting-a-fold-0060), or https://github.com/apache/netbeans/issues/6828, or https://github.com/skylot/jadx/issues/2073

Steps to Reproduce:
1. Open the "Open file" dialog (Using JFileChooser) in Windows L&F.
2. Navigate to a folder containing .exe files with the default Windows exe icon (e.g. `C:\windows`).

To finally fix the NPE change following line:

https://github.com/openjdk/jdk/blob/2e4a3c47e262f91a7f881d9d990eb81a929d1627/src/java.desktop/windows/classes/sun/awt/shell/Win32ShellFolder2.java#L1451

to:

~~~java
if (retVal != null && retVal.getWidth(null) != w) {
~~~

-------------

PR Comment: https://git.openjdk.org/jdk/pull/12010#issuecomment-1873980572