[jdk17u-dev] RFR: 8275320: NMT should perform buffer overrun checks

[Thomas Stuefe]

Not a clean downport:

- testutils.cpp/.hpp does not exist yet. I took this over (it is just a bunch of test helpers for gtests)
- os.cpp, in os::malloc, the size overflow check added with the downport of JDK-8286519 needed to be reformulated.

I also added a small, JDK17 only gtest to test for size argument overflow. Ideally that should have been done with JDK-8286519. It is release only since in debug we will get an assert which is fine but I did not feel like adding a death test for it. 

Note that in head, I am in the process of fixing https://bugs.openjdk.org/browse/JDK-8295865, and that will introduce much more thorough regression tests. Should we ever downport JDK-8295865 to 17, the test introduced here can be removed.

-------------

Commit messages:
 - Backport cf7adae6333c7446048ef0364737927337631f63

Changes: https://git.openjdk.org/jdk17u-dev/pull/842/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk17u-dev&pr=842&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8275320
  Stats: 570 lines in 11 files changed: 523 ins; 11 del; 36 mod
  Patch: https://git.openjdk.org/jdk17u-dev/pull/842.diff
  Fetch: git fetch https://git.openjdk.org/jdk17u-dev pull/842/head:pull/842

PR: https://git.openjdk.org/jdk17u-dev/pull/842