RFR: 8289697: buffer overflow in MTLVertexCache.m: MTLVertexCache_AddGlyphQuad [v4]

[Ajit Ghaisas]

On Wed, 6 Jul 2022 20:31:31 GMT, Vladimir Kempik <vkempik at openjdk.org> wrote:

>> Please review this simple patch. When running idea on jdk17 with asan I have found this buffer overflow.
>> The code checks the cache for at least one free element, while placing 6 elements to the cache.
>> The fix checks the presence of 6 free elements.
>
> Vladimir Kempik has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Change define name

The full test run is good with this change.
2 comments in the code still use `MTL_TRIS_IN_VERTEX`. These need to be corrected.

src/java.desktop/macosx/native/libawt_lwawt/java2d/metal/MTLVertexCache.m line 225:

> 223:                 maskCacheIndex);
> 224: 
> 225:     // MTLVC_ADD_TRIANGLES at the end of this function will place MTL_TRIS_IN_VERTEX vertexes to the vertex cache

Replace `MTL_TRIS_IN_VERTEX` from this comment.

src/java.desktop/macosx/native/libawt_lwawt/java2d/metal/MTLVertexCache.m line 313:

> 311:     J2dTraceLn(J2D_TRACE_INFO, "MTLVertexCache_AddGlyphQuad");
> 312: 
> 313:     // MTLVC_ADD_TRIANGLES adds MTL_TRIS_IN_VERTEX vertexes into Cache, so need to check space for MTL_TRIS_IN_VERTEX elements

Replace `MTL_TRIS_IN_VERTEX` from this comment.

-------------

PR: https://git.openjdk.org/jdk/pull/9368