RFR: 8289697: buffer overflow in MTLVertexCache.m: MTLVertexCache_AddGlyphQuad [v2]

[Vladimir Kempik]

On Tue, 5 Jul 2022 19:01:41 GMT, Phil Race <prr at openjdk.org> wrote:

> Are there any other cases like this ? We should look around. (PS I see Ajit asked the same question and perhaps even answered it !) Did you find this by inspection or did you hit it ? 

I hit this bug when running IDEA on asan-enabled build of ojdk

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6210011b94a8 at pc 0x0001707b9c0e bp 0x70001358e8f0 sp 0x70001358e8e8
WRITE of size 4 at 0x6210011b94a8 thread T56
    #0 0x1707b9c0d in MTLVertexCache_AddGlyphQuad+0x5ed (libawt_lwawt.dylib:x86_64+0x1cbc0d)
....
0x6210011b94a8 is located 8 bytes to the right of 4000-byte region [0x6210011b8500,0x6210011b94a0)
allocated by thread T56 here:
    #0 0x106857400 in wrap_malloc+0xa0 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x44400)
    #1 0x1707b8277 in MTLVertexCache_InitVertexCache+0x17 (libawt_lwawt.dylib:x86_64+0x1ca277)
>Can we use a defined constant instead of "6" ?

#define TRI_IN_VERT 6 ?

-------------

PR: https://git.openjdk.org/jdk/pull/9368