RFR: JDK-8068721 - RMI-IIOP communication fails when ConcurrentHashMap is passed to remote method

[Peter Levart]

On 03/23/2015 10:25 PM, Mark Sheppard wrote:
> Hi
>  please oblige and review the following fix
>
> http://cr.openjdk.java.net/~msheppar/8068721/jdk9/corba/webrev/
> http://cr.openjdk.java.net/~msheppar/8068721/jdk9/jdk/webrev/

Hi Mark,

Can an instance of ObjectStreamHook (ObjectOutputStream) be invoked from 
multiple threads concurrently? I don't see any synchronization elsewhere 
in ObjectStreamHook or ObjectOutputStream. Perhaps the synchronization 
in putFields()/writeFields() is not needed.

Also, in ObjecyOutputStream, writeFields() looks like this:

     public void writeFields() throws IOException {
         if (curPut == null) {
             throw new NotActiveException("no current PutField object");
         }
         bout.setBlockDataMode(false);
         curPut.writeFields();
         bout.setBlockDataMode(true);
     }


...meaning that NotActiveException is thrown in case user calls this 
method before invoking putFields() which initializes the PutField 
object. Should ObjectStreamHook behave the same?

Regards, Peter

>
>
> which addresses the issue in
> https://bugs.openjdk.java.net/browse/JDK-8068721
>
> This relates to RMI-IIOP and the interplay between custom marshalling 
> of ValueTypes and
> the corresponding serialization of a Java class, in this case 
> ConcurrentHashMap.
>
> ConcurrentHashMap changed its structure in jdk8 from that used in jdk7.
> This resulted in modification to the readObject and writeObject methods,
> and in particular, former serial fields were removed, resulting in
> writeObject using PutField and readObject using defaultReadObject.
> The writeObject invokes the putFields method of an ObjectOutputStream 
> multiple times, and assumes
> that it will receive the same PutField object instance for each 
> invocation. The spec
> doesn't endorse this behaviour - but that's what the implementation of 
> ObjectOutputStream
> provides. However in the case of RMI-IIOP, the OutputStreamHook, a 
> subclass of ObjectOutputStream, returns a new instance for each
> putFields invocation. Thus, the ConcurrentHashMap writeObject results 
> in improper serialization in the context
> of RMI-IIOP.
> In the unmarshalling flow of ConcurrentHashMap, the readObject now 
> uses defaultReadObject rather than GetField
> In this call flow the IIOPInputStream attempts to ignore any primitive 
> field, in the serialized data, that doesn't
> correspond with a reflective field of the object. However, it leaves 
> the primitive in the stream.
> Thus, in the case of ConcurrentHashMap, which has serialized two 
> integers and
> an array of Segments (subclass of ReentrantLock), this results in 
> erroneous
> deserialization, with a misinterpretation of a value tag in the stream 
> as an array length
> and an attempt to allocate a very large array ensues, with an 
> exception being thrown.
>
> The OutputStreamHook now returns the same instance of PutField 
> allocated for each separate call of putFields method.
> This highlights a need to tighten up and disambiguate the 
> OutputObjectStream spec for putFields.
>
> IIOPInputStream now removes the primitive values from the stream.
>
> regards
> Mark