Request for review: 6597112: Referential integrity loophole during remote object export

[Neil Richards]

Bug 6597112 [1] reports that the implementation of
java.rmi.server.UnicastRemoteObject.exportObject(Remote) has a window
in which the Remote object given to the method is only weakly
referenced by the implementation.

Due to this window, an unfortunately timed GC cycle can free up the
Remote object, causing an exception to be thrown at a later point
(still) within the exportObject() implementation.

In the implementation, the window occurs in
sun.rmi.server.UnicastServerRef.exportObject(), between the creation
of a sun.rmi.transport.Target object (which currently holds the object
being exported weakly), and the call to ref.exportObject(), which
retrieves a reference for the Remote from (the weak reference within)
the Target, completes the export of the Remote, storing the retrieved
(strong) reference in sun.rmi.transport.ObjectTable.

The weak reference within Target is actually an instance of
sun.rmi.transport.WeakRef (which extends java.lang.ref.WeakReference),
a package private class that is used only for storing refs within
Target objects, and (transiently) for performing lookups within the
ObjectTable.

WeakRef provides a facility for "pinning" (and unpinning) its object,
whereby it can hold a strong reference to the object (in addition to
the weak reference), thus preventing it (whilst pinned) from being
GC'd. Currently, a new WeakRef starts life "unpinned".

The attached suggested fix (in webrev zip file form) changes the
behaviour of WeakRef such that its reference is initally pinned. This
prevents the referred to (Remote) object from being initially GC-able,
until the call which places a (strong) reference to it in ObjectTable
has completed, at which point it can be unpinned.

In order to be able to unpin it at this point, the suggested fix
modifies the access modifier to Target.unpinImpl() (which calls onto
WeakRef.unpin()) from 'default' (ie. package private) to 'public'.
(This is necessary as the caller, UnicastServerRef, is in a different
package to Target).

Also included is a testcase which is likely to demonstrate the
problem, based on the code given in the original bug submission.

Without the fix, on my machine (a Linux x86 dual Pentium 4 - gasp at
the sheer *power*), the problem is seen within around 5000 iterations,
so I've set the maximum iteration count (ie. the point at which the
test concludes the problem doesn't exist) to be 10x that.

Please review the suggested fix attached, and reply with your
comments/suggestions,
Thanks,
Neil

--
Unless stated above:
IBM email: neil_richards at uk.ibm.com
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU