RFR: 8293862: javax/swing/JFileChooser/8046391/bug8046391.java failed with 'Cannot invoke "java.awt.Image.getWidth(java.awt.image.ImageObserver)" because "retVal" is null'

[Alexey Ivanov]

This is somewhat a continuation of #11104 where the issue was discussed and where I found [the root cause](https://github.com/openjdk/jdk/pull/11104#issuecomment-1382435784).

**Root Cause**

The icon extraction code compares the returned handle to zero: 
https://github.com/openjdk/jdk/blob/8cb25d3de494c6d9357a1c199e1a9dbff9be9948/src/java.desktop/windows/classes/sun/awt/shell/Win32ShellFolder2.java#L1157-L1158

On 64-bit systems, a valid handle can be negative. For example:


getIcon : Desktop 16(16)
GetIconLocation(uFlags=0x22, flags=0x4, index=-110) SUCCESS - szBuf=C:\Windows\system32\imageres.dll
Extract - hres: 0, hIcon=0000000019B200F9, hIconSmall=FFFFFFFFF98C00EB, size=16(0x100020)
SUCCEEDED
!!! hIcon(0xfffffffff98c00eb) <= 0 : Desktop 16(16)


Here in `size=16(0x100020)`, 16 is the requested icon size, the value in parenthesis is value of `iconSize` passed to the [`Extract`](https://learn.microsoft.com/en-us/windows/win32/api/shlobj_core/nf-shlobj_core-iextracticonw-extract) method that contains the size for small icon in high word and that for large icon in low word. `Desktop 16(16)` means the icon of size 16×16 is requested, and we're extracting 16×16 icon.

The icon was extracted _successfully_ but its handle was interpreted as an error.

The same problem exists in the fallback code:
https://github.com/openjdk/jdk/blob/8cb25d3de494c6d9357a1c199e1a9dbff9be9948/src/java.desktop/windows/classes/sun/awt/shell/Win32ShellFolder2.java#L1161

Then when the fallback code is executed, another problem occurs:


GetIconLocation(uFlags=0x40, flags=0x1a3, index=-675088752) SUCCESS - szBuf=
Extract - hres: 80004005, hIcon=0000000000000000, hIconSmall=000001A3D7C2F83A, size=16(100020)
NOT SUCCEEDED
hIcon = 0x1a3d7c2f83a : Desktop 16(16)


The error code 0x80004005 is `E_FAIL`. Yet the return value is not verified in the code:
https://github.com/openjdk/jdk/blob/8cb25d3de494c6d9357a1c199e1a9dbff9be9948/src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp#L997-L1003

Since the icon handle is *seemingly* valid, it continues to extracting the icon bits:


native getIconBits - fn_GetIconInfo returned false
Error code: 0x57a - Invalid cursor handle.
makeIcon: iconBits = null


It's not a surprise. After all, the `Extract` method returned an error, which means neither icon handle is valid.

In the end, the `MultiResolutionIcon` contains `null`:

    16 -> null
    32 -> [BufferedImage at 59e735d](mailto:BufferedImage at 59e735d)
    24 -> [BufferedImage at 60b2faf0](mailto:BufferedImage at 60b2faf0)


This problem has existed for since [JDK-8182043](https://bugs.openjdk.org/browse/JDK-8182043), yet it went unnoticed because `getResolutionVariant` just returned a null. However, after [JDK-8282526](https://bugs.openjdk.org/browse/JDK-8282526) was integrated, the result is used, which leads to `NullPointerException`.


**The Fix**

Handle the errors as `hIcon == E_PENDING || hIcon == 0` and `hIcon == 0`. In the fallback code where a default icon is requested, [`E_PENDING` cannot be returned](https://learn.microsoft.com/en-us/windows/win32/api/shlobj_core/nf-shlobj_core-iextracticonw-geticonlocation#remarks), it's valid only if `GIL_ASYNC` flag is used.

Add `assert` statements to `MultiResolutionIconImage` for null-checks which will help us to catch a similar situation if it occurs.

**Testing**

The two errors combined lead to random failures. No particular host reproduced the problem more often.

While I was testing with the additional traces, sometimes 5 of 10 tests failed, other times only 1 of 10. There were a couple of test runs where no failures occurred.

With the fix applied, no failures occurred.

-------------

Commit messages:
 - 8293862: javax/swing/JFileChooser/8046391/bug8046391.java failed with 'Cannot invoke "java.awt.Image.getWidth(java.awt.image.ImageObserver)" because "retVal" is null'

Changes: https://git.openjdk.org/jdk/pull/12010/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=12010&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8293862
  Stats: 29 lines in 4 files changed: 17 ins; 2 del; 10 mod
  Patch: https://git.openjdk.org/jdk/pull/12010.diff
  Fetch: git fetch https://git.openjdk.org/jdk pull/12010/head:pull/12010

PR: https://git.openjdk.org/jdk/pull/12010