RFR: 8296496: Overzealous check in sizecalc.h prevents large memory allocation [v3]

[Sergey Bylokhov]

On Tue, 8 Nov 2022 23:43:23 GMT, Alexander Zuev <kizune at openjdk.org> wrote:

>> Removed the additional multiplication overflow detection.
>> Instead cast all the parameters to type_t just the way they are treated in the existing size check macro. 
>> This way there is no possibility to accidentally provide parameters that will pass the size check macro while being cast to size_t there but then due to the missing cast cause the wrong size passed the actual allocation function.
>> Since this checking macro was used in couple of different places all of them needs to be updated in the similar way.
>
> Alexander Zuev has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fixed casting in SAFE_SIZE_NEW_ARRAY2

src/java.desktop/share/native/common/awt/utility/sizecalc.h line 94:

> 92: 
> 93: #define SAFE_SIZE_NEW_ARRAY2(type, n, m) \
> 94:     (IS_SAFE_SIZE_MUL((m), (n)) && IS_SAFE_SIZE_MUL(sizeof(type), (n) * (m)) ? \

Why we do not cast it here: (n) * (m)?

-------------

PR: https://git.openjdk.org/jdk/pull/11030