RFR: 8289697: buffer overflow in MTLVertexCache.m: MTLVertexCache_AddGlyphQuad

[Ajit Ghaisas]

On Mon, 4 Jul 2022 15:40:56 GMT, Vladimir Kempik <vkempik at openjdk.org> wrote:

> Please review this simple patch. When running idea on jdk17 with asan I have found this buffer overflow.
> The code checks the cache for at least one free element, while placing 6 elements to the cache.
> The fix checks the presence of 6 free elements.

This is a good catch and fix.

There is another place in this file where `MTLVC_ADD_TRIANGLES` is used.
Although it is a corner case, it is worth adding the check for additional 6 vertices at that place as well.
Adding a separate check at the beginning of the `MTLVertexCache_AddMaskQuad` method where there is a check for `maskCacheIndex` seems logical to me. What do you think?

-------------

PR: https://git.openjdk.org/jdk/pull/9368