RFR: 8372589: VM crashes on init when NonNMethodCodeHeapSize is set too small and UseTransparentHugePages is enabled

[Martin Doerr]

On Thu, 22 Jan 2026 10:01:17 GMT, Martin Doerr <mdoerr at openjdk.org> wrote:

>> @dbriemann,  this change invalidated assumption that ReservedCodeCacheSize can't change if specified on command line. You replaced `align_down` with `align_up` but did not check that `cache_size` may increase after that.
>> 
>> It also causing issue with AOT because CodeCache size varies between different phases because we use different number of compiler threads and as result different NonNmethod section size.
>
> @vnkozlov: If `ReservedCodeCacheSize` is specified on the command line, but `NonProfiledCodeHeapSize` or `ProfiledCodeHeapSize` is not specified explicitly, we could subtract from one of them. What do you think about that?

> @TheRealMDoerr current code at https://github.com/openjdk/jdk/blob/master/src/hotspot/share/code/codeCache.cpp#L228-L262 is doing exactly that. But new code from these changes at lines L305-L308 invalidates that by aligned up sizes and recalculates new code cache size. Previous code aligned down sizes but did not recalculate code cache size and, as result, sum of sections sizes could be smaller then it. Which is fine except this bug. I am not against aligning up but we need to do it before code cache size checks and be smarter how we align it. We can adjust default ReservedCodeCacheSize so that aligning up will not change it.

I mean subtracting it after aligning. Something like 

diff --git a/src/hotspot/share/code/codeCache.cpp b/src/hotspot/share/code/codeCache.cpp
index 95a2fb908de..e03efdac3ae 100644
--- a/src/hotspot/share/code/codeCache.cpp
+++ b/src/hotspot/share/code/codeCache.cpp
@@ -305,7 +305,13 @@ void CodeCache::initialize_heaps() {
   non_nmethod.size = align_up(non_nmethod.size, min_size);
   profiled.size = align_up(profiled.size, min_size);
   non_profiled.size = align_up(non_profiled.size, min_size);
-  cache_size = non_nmethod.size + profiled.size + non_profiled.size;
+  if (FLAG_IS_CMDLINE(ReservedCodeCacheSize) && !FLAG_IS_CMDLINE(NonProfiledCodeHeapSize)) {
+    non_profiled.size = cache_size - non_nmethod.size - profiled.size;
+  } else if (FLAG_IS_CMDLINE(ReservedCodeCacheSize) && !FLAG_IS_CMDLINE(ProfiledCodeHeapSize)) {
+    profiled.size = cache_size - non_nmethod.size - non_profiled.size;
+  } else {
+    cache_size = non_nmethod.size + profiled.size + non_profiled.size;
+  }
 
   FLAG_SET_ERGO(NonNMethodCodeHeapSize, non_nmethod.size);
   FLAG_SET_ERGO(ProfiledCodeHeapSize, profiled.size);


I think align_down is not ideal when large pages are used and the sizes are tuned by the user. We should have them at least as large as specified.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28658#issuecomment-3785704328