RFR(10): JDK-8178536: OOM ERRORS + SERVICE-THREAD TAKES A PROCESSOR TO 100%
mandy.chung at oracle.com
Wed Jun 14 17:18:04 UTC 2017
> On Jun 7, 2017, at 8:53 AM, Poonam Parhar <poonam.bajaj at oracle.com> wrote:
> Could I have reviews for the changes for the Bug JDK-8178536 <https://bugs.openjdk.java.net/browse/JDK-8178536>: OOM ERRORS + SERVICE-THREAD TAKES A PROCESSOR TO 100%.
> Problem: If there are listeners installed for MemoryMXBean then in the event of an OOME, JVM internal thread 'Service Thread' responsible for delivering notifications to the Java threads itself may encounter an OOM exception and get into a loop of processing its pending requests. This happens if and when the Service Thread executing the native code calls its corresponding java methods and faces an OOM exception, this pending exception makes the thread exit early from SensorInfo∷trigger() function before it can update its pending requests counter (_pending_trigger_count). This pending exception is never cleared and that makes the thread loop in LowMemoryDetector::process_sensor_changes().
> Hotspot changes:
> http://cr.openjdk.java.net/~poonam/8178536/webrev.hotspot/ <http://cr.openjdk.java.net/~poonam/8178536/webrev.hotspot/>
> These changes check for the pending exception and clear it, and make sure that the pending requests counters are kept in sync on both the Java and the VM side.
This fix would drop any notification when it runs out of memory. This approach is reasonable. Even we preallocate the notification object, the listener code may fail due to OOM.
There are two places doing the object allocation - VM allocates MemoryUsage and the Sensor::trigger implementation. Another idea is to change Sensor::trigger to take the values of init, used, committed, max instead MemoryUsage so that the VM code can be simplified and the Java code will catch all exception and return a boolean to indicate success/fail. This will also prepare moving the invocation of the user listener code in a Java thread instead of ServiceThread to address the issue about ServiceThread is a hidden thread that can’t be debugged.
> JDK changes:
> http://cr.openjdk.java.net/~poonam/8178536/webrev.jdk/ <http://cr.openjdk.java.net/~poonam/8178536/webrev.jdk/>
> These changes make the triggerAction() a no-op since we need to call this method if MemoryService::create_MemoryUsage_obj() encounters an OOM exception and we want to avoid further potential OOM exceptions in triggerAction(MemoryUsage).
> I have written a testcase though I won't be integrating it along with the fix because the failure is unreliable; the Service Thread does not always fall into the situation that causes it to loop i.e. it has _pending_trigger_count>0 and hits an OOM in its Java calls.
> http://cr.openjdk.java.net/~poonam/8178536/webrev.hotspot.testcase/ <http://cr.openjdk.java.net/~poonam/8178536/webrev.hotspot.testcase/>
More information about the hotspot-runtime-dev