RFR: JDK-8312182: THPs cause huge RSS due to thread start timing issue

[Thomas Stuefe]

On Wed, 19 Jul 2023 04:25:26 GMT, David Holmes <dholmes at openjdk.org> wrote:

> Wow what a mess! The OS is really working against us in this area. We should not have to be playing these silly games.
> 
> There is a caution in `os::Linux::default_guard_size` that glibc guard pages are expensive to create so what are we paying for this?

Less, actually :-) The patch is a performance win too.

Code-wise, glibc seems to do one more `mmap(2)` call if we use guard pages, at least for the standard code paths. 

I wanted to see if that matters, so I tested with THP mode set to "always" on my 48-core Linux 5.15 glibc 2.31 box. I tested with two stack sizes (1 MB, 2 MB), with and without patch. The test measures the time it takes to spawn 10000 threads.

| Stack size (kb)|  with patch (ms) | without patch (ms) | Delta |
|----------------|------------------|--------------------|-------|
| 1024           |  723 +- 11       |   807 +- 18        | +12%  |
| 2048           |  710 +- 13       |  1621 +- 91        | +128% |

The numbers are consistently better with my patch. When the stack size approaches THP page size, the difference gets *huge*!

This makes sense since `khugepaged` transforms thread pages under our noses while child threads, freshly started, try to use the same pages. So we get a lot of interference. The closer the stack size is to THP size, the more likely this is. And we use more memory too, which costs bandwidth.

I repeated the test with THPs disabled, and sure enough, the difference vanished.

Bottomline, the cost of the additional mmap is dwarfed by the win of not having to fight khugepaged for the thread stacks.

> I don't think these two adjustments should be combined like this. The existing +1 may be sufficient for some folk who do not want the cost of creating a glibc guard page as well.

I disagree; I'd like to enable this patch by default as a whole if we run with THP=always; we save performance and memory. There are not many downsides.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14919#issuecomment-1641562585