Analysis on JDK-8022321 java/lang/ref/OOMEInReferenceHandler.java fails intermittently

[Kim Barrett]

> On Mar 23, 2016, at 10:02 AM, Peter Levart <peter.levart at gmail.com> wrote:
> ...so I checked what it would be needed if there was such getPendingReferences() native method. It turns out that a single native method would not be enough to support the precise direct ByteBuffer allocation. Here's a refactored webrev that introduces a getPendingReferences() method which could be turned into a native equivalent one day. There is another native method needed - int awaitEnqueuePhaseStart():
> 
> http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.09.part2/

I don't think the Reference.awaitEnqueuePhaseStart thing is needed.

Rather, I think the Direct-X-Buffer allocation should conspire with
the the Direct-X-Buffer cleanups directly to manage that sort of
thing, and not add anything to Reference and the reference processing
thread.  E.g. the phase and signal/wait are purely part of
Direct-X-Buffer.  (I also think something like that could/should have
been done instead of providing Direct-X-Buffer with access to
Reference.tryHandlePending, but that's likely water under the bridge
now.)

Something very roughly like this:

allocating thread, after allocation failed

bool waitForCleanups() {
  int epoch = DXB.getCleanupCounter();
  long start = startTime();
  long timeout = calcTimeout(start)
  synchronized (DXB.getCleanupMonitor()) {
    while (epoch == DBX.getCleanupCounter()) {
      wait(timeout);
      timeout = calcTimeout(start);
      if (timeout <= 0) break;
    }
    return epoch != DBX.getCleanupCounter();
  }
}

cleanup function, after freeing memory

  synchronized (DBX.getCleanupMonitor()) {
    DBX.incCleanupCounter();
    DBX.getCleanupMonitor().notify_all();
  }

Actually, epoch should probably have been obtained *before* the failed
allocation attempt, and should be an argument to waitForCleanups.

That's all quite sketchy, but I need to do other things today.

Peter, care to try filling this in?