RFR(xs): 8155211: Ucrypto Library leaks native memory

[Thomas Stüfe]

Hi all,

could you please review this small fix?

Webrev:
http://cr.openjdk.java.net/~stuefe/webrevs/8155211-ucrypto-lib-mem-leak/webrev.00/webrev/

Bug: https://bugs.openjdk.java.net/browse/JDK-8155211

Basically, during benchmarks our Solaris VMs went out-of-native-memory;
when we investigated, we saw ~31million small lost allocations from
nativeCrypto.c:663. Looking at the source shows that the output buffer is
not deallocated if outLen is 0.

outLen can be 0 if the java output buffer was len 0 to begin with, or if
the outOfs equals output array len, or if CipherFinal returns 0. In the
first two cases we will call calloc(0), which on Solaris is implemented as
a real calloc (returns a unique pointer which needs to be freed).

I changed the coding to always free the internal buffers. I also removed
the (bufOut != NULL) condition from SetByteArrayRegion, because that should
be always true at this point.

Kind Regards, Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/security-dev/attachments/20160428/a2c5cb94/attachment.htm>