RFR: JDK-8285712: LogMessageBuffer doesn't check vsnprintf return value

[David Holmes]

On Wed, 27 Apr 2022 10:34:53 GMT, Johan Sjölén <duke at openjdk.java.net> wrote:

> os::vsnprintf can return a negative value on encoding error. A negative return value will cause wraparound when cast to size_t. This in turn causes LogMessageBuffer::grow() to attempt a large memory allocation. Instead of accepting this we bail on the logging.

Hi Johan,

Looks good. One small suggested change.

Thanks,
David

src/hotspot/share/logging/logMessageBuffer.cpp line 117:

> 115:     if (ret < 0) {
> 116:       // Encoding error occurred, bail.
> 117:       va_end(copy);

To avoid duplication I suggest moving the `va_end` to immediately after `os::vsnprintf`.

-------------

Changes requested by dholmes (Reviewer).

PR: https://git.openjdk.java.net/jdk/pull/8423