RFR: 8296496: Overzealous check in sizecalc.h prevents large memory allocation [v2]

[Alexander Zuev]

On Tue, 8 Nov 2022 22:37:09 GMT, Alexey Ivanov <aivanov at openjdk.org> wrote:

>>> Each parameter must be cast as in SAFE_SIZE_ARRAY_ALLOC.
>> 
>> If we do that then logic might be broken since we checking for the limits against the (size_t)(m * n) but performing call with ((size_t)(m) * (size_t)(n)) which might add potential point of failure. I prefer to do the conversions in the same way we do them in checks.
>
> It's [what you just did](https://github.com/openjdk/jdk/pull/11030/files#diff-f4ac4382758a5b83444f9483cbc756ca6572fddf4c3811f2f5d86002a91599c2L77-R74) for `SAFE_SIZE_ARRAY_ALLOC`:
> 
>  #define SAFE_SIZE_ARRAY_ALLOC(func, m, n) \
> -    (IS_SAFE_SIZE_MUL((m), (n)) ? ((func)((m) * (n))) : FAILURE_RESULT)
> +    (IS_SAFE_SIZE_MUL((m), (n)) ? ((func)((size_t)(m) * (size_t)(n))) : FAILURE_RESULT)
> 
> You cast both `m` and `n`.
> 
> This code basically does the same. If you don't cast each parameter, the result may be different, since there are cases where `((size_t)(m) * (size_t)(n))` is not equal to `(size_t)((m) * (n))`.

Ok, i see what you mean. Ok, that might work since we are already checking SAFE_SIZE_MUL((m), (n)) in the line above.

-------------

PR: https://git.openjdk.org/jdk/pull/11030