MetaspaceGC::_capacity_until_GC exceeds MaxMetaspaceSize

[Yasumasa Suenaga]

Hi all,

I'm Yasumasa Suenaga (OpenJDK reviewer: ysuenaga)

My customer uses OpenJDK 8u131 with following commandline arguments:

   -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=80
   -XX:CompressedClassSpaceSize=128m -Xms4500m -Xmx4500m
   -XX:MaxMetaspaceSize=256m -Xmn=768m -XX:MaxTenuringThreshold=15
   -XX:OnOutOfMemoryError="/bin/kill -ABRT %p"
   -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseConcMarkSweepGC

Their system has encountered OutOfMemoryError which is caused by Metaspace.

I checked core image which was collected by OnOutOfMemoryError, and I saw
strange values in it as below:

-------------------
(gdb) p Metaspace::_space_list->_reserved_words
$67 = 31195136
(gdb) p Metaspace::_class_space_list->_reserved_words
$68 = 16777216
(gdb) p MetaspaceGC::_capacity_until_GC
$75 = 448045056
-------------------

MaxMetaspaceSize and CompressedClassSpaceSize seem to affect Metaspace.
However, MetaspaceGC::_capacity_until_GC exceeds MaxMetaspaceSize.
Is it correct?

I checked used and committed memory from the core, they seem to be enough
to allocate new objects as below:

-------------------
(gdb) p MetaspaceAux::_used_words
$62 = {1742673, 17905981}
(gdb) p Metaspace::_space_list->_committed_words
$64 = 30932992
(gdb) p Metaspace::_class_space_list->_committed_words
$65 = 2621440
-------------------

I guess the cause of OOME is invalid value in _capacity_until_GC.
_capacity_until_GC is used for expanding / shrinking Metaspace.
However it is not checked when it changes.

So I think we need to add the code to check it as below:
(The change is for jdk/jdk)
-------------------
diff -r 32c6cc430526 src/hotspot/share/memory/metaspace.cpp
--- a/src/hotspot/share/memory/metaspace.cpp    Wed Jan 09 22:59:49 2019 +0100
+++ b/src/hotspot/share/memory/metaspace.cpp    Thu Jan 10 17:56:55 2019 +0900
@@ -143,6 +143,10 @@
      new_value = align_down(max_uintx, Metaspace::commit_alignment());
    }
  
+  if (new_value > MaxMetaspaceSize) {
+    return false;
+  }
+
    size_t prev_value = Atomic::cmpxchg(new_value, &_capacity_until_GC, old_capacity_until_GC);
  
    if (old_capacity_until_GC != prev_value) {
-------------------

_capacity_until_GC is set to MaxMetaspaceSize at MetaspaceGC::initialize(),
and it set to committed size or MetaspaceSize at MetaspaceGC::post_initialize().
So I guess max value of it is MaxMetaspaceSize.


What do you think?
If it is a bug, I will file JBS and send review request.


Thanks,

Yasumasa