RFR: 8277299: STACK_OVERFLOW in Java_sun_awt_shell_Win32ShellFolder2_getIconBits

[Sergey Bylokhov]

On Fri, 19 Nov 2021 20:47:01 GMT, Alexey Ivanov <aivanov at openjdk.org> wrote:

>> src/java.desktop/windows/native/libawt/windows/ShellFolder2.cpp line 1060:
>> 
>>> 1058:             int nBits = iconSize * iconSize;
>>> 1059:             long * colorBits;
>>> 1060:             colorBits = (long*)safe_Malloc(MAX_ICON_SIZE * MAX_ICON_SIZE * sizeof(long));
>> 
>> I am not sure that the bad_alloc will be properly handled in this Java_sun_awt_shell_Win32ShellFolder2_getIconBits method.
>> +Probably it will be better to merge assigning into one line.
>
>> I am not sure that the bad_alloc will be properly handled in this Java_sun_awt_shell_Win32ShellFolder2_getIconBits method.
> 
> I can't see any try-catch.
> Is it better to use `malloc` and check for `NULL`?
> 
>> Probably it will be better to merge assigning into one line.
> 
> I agree. The line doesn't get too long.

Probably TRY/CATCH_BAD_ALLOC_RET(NULL); could be used. It depends on how we will clean the stuff on exception.

-------------

PR: https://git.openjdk.java.net/jdk/pull/6473