Large page use crashes the JVM on some Linux systems

[B. Blaser]

On 2 May 2018 at 10:38, B. Blaser <bsrbnd at gmail.com> wrote:
> Hi Claes,
>
> On 24 April 2018 at 21:39, Claes Redestad <claes.redestad at oracle.com> wrote:
>> The root issue here could very well be that the SHM sanity test is
>> insufficient. Adding the same test as we already do for TLBFS seems like the
>> wrong approach.
>>
>> I'm not the most knowledgeable about SHM, though, in fact not knowledgeable
>> at all, so let's try and get you subscribed to hotspot-dev and spark a
>> discussion on the list.
>>
>> /Claes

Experimenting on different systems, I note that the following test (
'shmget(SHM_HUGETLB) == -1' ) isn't always reliable:

http://hg.openjdk.java.net/jdk/jdk/file/35b22ca681d1/src/hotspot/os/linux/os_linux.cpp#l3709

It may happen that small pages are used when no huge pages are
available resulting in a further JVM crash due to a
non-huge-page-size-alignment of the allocated memory, as discussed
previously.

It seems that the test above could be enforced by verifying that 'size
<= (free+surplus-reserved)*hugepagesize' according to [1] and [2].

Here is a suggested fix for that, tier1 is OK (with 'HugePages_Total:
0') but any further testing on different configurations/systems would
be very valuable.

What do you think?

Thanks,
Bernard

[1] https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
[2] $ man 5 proc

diff -r e81481fea884 src/hotspot/os/linux/os_linux.cpp
--- a/src/hotspot/os/linux/os_linux.cpp    Mon May 07 08:56:35 2018 +0200
+++ b/src/hotspot/os/linux/os_linux.cpp    Wed May 09 19:10:32 2018 +0200
@@ -3617,6 +3617,37 @@
     shm_warning_format(str " (error = %d)", err);  \
   } while (0)

+static bool shm_available(size_t bytes) {
+  size_t free = SIZE_MAX, rsvd = SIZE_MAX, surp = SIZE_MAX;
+  char line[40];
+
+  FILE *fp = fopen("/proc/meminfo", "r");
+  if (fp) {
+    while ((free == SIZE_MAX || rsvd == SIZE_MAX || surp == SIZE_MAX)
&& !feof(fp)) {
+      fgets(line, sizeof(line), fp);
+
+      if (free == SIZE_MAX && sscanf(line, "HugePages_Free: %zu", &free) == 1)
+        continue;
+
+      if (rsvd == SIZE_MAX && sscanf(line, "HugePages_Rsvd: %zu", &rsvd) == 1)
+        continue;
+
+      if (surp == SIZE_MAX && sscanf(line, "HugePages_Surp: %zu", &surp) == 1)
+        continue;
+    }
+    fclose(fp);
+  }
+
+  gid_t gid = -1;
+  if (fp = fopen("/proc/sys/vm/hugetlb_shm_group", "r")) {
+    fscanf(fp, "%u", &gid);
+    fclose(fp);
+  }
+
+  return (free+surp-rsvd)*os::large_page_size() >= bytes
+    && (!geteuid() || group_member(gid));
+}
+
 static char* shmat_with_alignment(int shmid, size_t bytes, size_t alignment) {
   assert(is_aligned(bytes, alignment), "Must be divisible by the alignment");

@@ -3703,6 +3734,11 @@
     return NULL; // Fallback to small pages.
   }

+  if (!shm_available(bytes)) {
+    shm_warning("Shared memory not available.");
+    return NULL;
+  }
+
   // Create a large shared memory region to attach to based on size.
   // Currently, size is the total size of the heap.
   int shmid = shmget(IPC_PRIVATE, bytes, SHM_HUGETLB|IPC_CREAT|SHM_R|SHM_W);