RFR: 8373626: [asan] read past end of buffer in sun.awt.image.ImagingLib.convolveBI [v5]

[Sergey Bylokhov]

On Mon, 23 Feb 2026 20:27:48 GMT, Phil Race <prr at openjdk.org> wrote:

>> Some of the medialib native functions implementing Convolve read data from arrays when it is not needed or used instead of reading just what is needed and used.
>> This is detected as a read out of bounds. It is limited and hasn't been seen to result in any crashes without ASAN, and the OOB values that are read are never used so there's a very limited problem.
>> The changes here make the mlib_ImageConv_*nw.c files match what happens in the mlib_ImageConv_*ext.c files which read just the data they need.
>> The changes are fairly mechanical but there could be copy/paste errors for a reviewer to find.
>> 
>> Not easy to provide a test case, building with --enable-asan is needed and for me it works only on macOS.
>> I did that and ran all our existing automated tests on our CI systems.
>
> Phil Race has updated the pull request incrementally with one additional commit since the last revision:
> 
>   8373626

Changes requested by serb (Reviewer).

src/java.desktop/share/native/libmlib_image/mlib_ImageConv_16nw.c line 1093:

> 1091: 
> 1092:             k0 = pk[0]; k1 = pk[1]; k2 = pk[2];
> 1093: 

`sp += (kw - 1)*chan1` is missing?
it exists in kw == 6,5,4,2,1 and only missing here for kw == 3

-------------

PR Review: https://git.openjdk.org/jdk/pull/29257#pullrequestreview-3856546242
PR Review Comment: https://git.openjdk.org/jdk/pull/29257#discussion_r2855103572