RFR: 8371647: 7 Integer overflows in mlib_malloc of mlib_sys.c:85

[Phil Race]

On Fri, 28 Nov 2025 22:02:09 GMT, Damon Nguyen <dnguyen at openjdk.org> wrote:

> There is a possible overflow when using `mlib_alloc()`. For example, `mlib_alloc(sizeof(mlib_s32) * (m * n))` may overflow if m and n are greater than 46430, since this would be greater than the max value for a signed 32 bit integer. I have added `SAFE_TO_ADD` and `SAFE_TO_MULT` in an attempt to amend this issue. CI testing shows all green.

src/java.desktop/share/native/libmlib_image/mlib_ImageConv_16ext.c line 271:

> 269:     if (!SAFE_TO_MULT(bsize, (mlib_s32)sizeof(FTYPE))) return MLIB_FAILURE;
> 270: 
> 271:     pbuff = mlib_malloc(sizeof(FTYPE)*bsize);

If mlib_malloc ends up in 
void *__mlib_malloc(mlib_u32 size); 
which I think it must do, because I can't find anything else,
then that accepts an unsigned 32 bit int, which makes sense because malloc accepts a size_t which is unsigned.

Note that sizeof() returns size_t too, so the multiplication result should be promoted to unsigned in the existing code, and preserved when passed as an arg.

But SAFE_TO_MULT will return a failure on overflow of signed arithmetic.  So I think we need something different here so we don't reject cases which are actually OK. ie in at least cases like this, we want to detect overflow of 32 bit unsigned, not 32 bit signed.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28560#discussion_r2578328918