RFR: 8296507: GCM using more memory than necessary with in-place operations [v3]

[Jamil Nimeh]

On Thu, 1 Dec 2022 04:19:37 GMT, Anthony Scarpino <ascarpino at openjdk.org> wrote:

>> I would like a review of an update to the GCM code.  A recent report showed that GCM memory usage for TLS was very large.  This was a result of in-place buffers, which TLS uses, and how the code handled the combined intrinsic method during decryption.  A temporary buffer was used because the combined intrinsic does gctr before ghash which results in a bad tag.  The fix is to not use the combined intrinsic during in-place decryption and depend on the individual GHASH and CounterMode intrinsics.  Direct ByteBuffers are not affected as they are not used by the intrinsics directly.
>> 
>> The reduction in the memory usage boosted performance back to where it was before despite using slower intrinsics (gctr & ghash individually).  The extra memory allocation for the temporary buffer out-weighted the faster intrinsic.
>> 
>> 
>>     JDK 17:   122913.554 ops/sec
>>     JDK 19:    94885.008 ops/sec
>>     Post fix: 122735.804 ops/sec 
>> 
>> There is no regression test because this is a memory change and test coverage already existing.
>
> Anthony Scarpino has updated the pull request incrementally with one additional commit since the last revision:
> 
>   comment update

This looks good, I only have nit comments.

src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 580:

> 578:      * an upper limit on the number of blocks encrypted in the intrinsic.
> 579:      *
> 580:      * For decrypting in-place byte[], calling methods must ct must set to null

Typo nit?  Should it be "calling methods must set ct to null"

src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java line 1642:

> 1640:                 // Clear output data
> 1641:                 dst.reset();
> 1642:                 // If this is no an in-place array, zero the dst buffer

nit: no -> not

-------------

Marked as reviewed by jnimeh (Reviewer).

PR: https://git.openjdk.org/jdk/pull/11121