[jdk11u-dev] RFR: 8312182: THPs cause huge RSS due to thread start timing issue [v2]

[Thomas Stuefe]

> Unclean composite backport to fix JDK-8312182 - "THPs cause huge RSS due to thread start timing issue" (https://bugs.openjdk.org/browse/JDK-8312182)
> 
> Problem:
> 
> On a machine with transparent huge pages (THP) unconditionally enabled (/sys/kernel/mm/transparent_hugepage/enabled = "always"), the JVM may show a huge memory footprint (RSS) and degraded thread start performance.
> 
> The following factors make the problem more severe and more likely:
> - thread stack size of 2M (on arm64 or x64) or larger
> - many threads, or high thread creation churn
> - a slow or overloaded machine (since part of the problem is timing-dependent)
> 
> For a detailed discussion of the underlying problem, please see https://github.com/openjdk/jdk/pull/14919.
> 
> ----------------
> 
> In jdk Head, the issue got fixed with a sequence of patches:
> 
> - [JDK-8303215](https://bugs.openjdk.org/browse/JDK-8303215) "Make thread stacks not use huge pages"
> - [JDK-8312182](https://bugs.openjdk.org/browse/JDK-8312182) "THPs cause huge RSS due to thread start timing"
> 
> However, JDK-8312182 itself needed one preparatory fix:
> - [JDK-8310233](https://bugs.openjdk.org/browse/JDK-8310233) "Fix THP detection on Linux"
> 
> and then we had several corner-case test problems which are fixed with:
> - [JDK-8312394](https://bugs.openjdk.org/browse/JDK-8312394) "[linux] SIGSEGV if kernel was built without hugepage support"
> - [JDK-8312620](https://bugs.openjdk.org/browse/JDK-8312620) "WSL Linux build crashes after JDK-8310233"
> - [JDK-8314139](https://bugs.openjdk.org/browse/JDK-8314139) "TEST_BUG: runtime/os/THPsInThreadStackPreventionTest.java could fail on machine with large number of cores"
> 
> and finally, we decided to rename the switch that allows to switch off the THP mitigation with a final patch:
> - [JDK-8312585](https://bugs.openjdk.org/browse/JDK-8312585) "Rename DisableTHPStackMitigation flag to THPStackMitigation"
> 
> 
> 
> Instead of downporting these 7 patches verbatim, I prepared a composite patch containing only the necessary mitigation and mitigation tests.
> 
> This patch does:
> - make sure that all thread stacks have at least one glibc guard page to prevent clustering of adjacent thread stacks into one VMA
> - change the default size of stacks to be not aligned to 2MB to prevent intra-stack THPs from forming
> 
> The patch needs some infrastructure, but I downported only the necessary parts: the helper class "HugePages", which is used in head to scan the operating system for information about THP settings. I only included the parts to do...

Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains two additional commits since the last revision:

 - Merge branch 'openjdk:master' into tstuefe-backport-84b325b8
 - Backport 84b325b844c08809448a9c073a11443d9e3c3f8e

-------------

Changes:
  - all: https://git.openjdk.org/jdk11u-dev/pull/2086/files
  - new: https://git.openjdk.org/jdk11u-dev/pull/2086/files/c13269c1..3709c1e6

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk11u-dev&pr=2086&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk11u-dev&pr=2086&range=00-01

  Stats: 30555 lines in 347 files changed: 9325 ins; 10833 del; 10397 mod
  Patch: https://git.openjdk.org/jdk11u-dev/pull/2086.diff
  Fetch: git fetch https://git.openjdk.org/jdk11u-dev.git pull/2086/head:pull/2086

PR: https://git.openjdk.org/jdk11u-dev/pull/2086