RFR: JDK-8068721 - RMI-IIOP communication fails when ConcurrentHashMap is passed to remote method

[Mark Sheppard]

Hi
  please oblige and review the following fix

http://cr.openjdk.java.net/~msheppar/8068721/jdk9/corba/webrev/
http://cr.openjdk.java.net/~msheppar/8068721/jdk9/jdk/webrev/


which addresses the issue in
https://bugs.openjdk.java.net/browse/JDK-8068721

This relates to RMI-IIOP and the interplay between custom marshalling of 
ValueTypes and
the corresponding serialization of a Java class, in this case 
ConcurrentHashMap.

ConcurrentHashMap changed its structure in jdk8 from that used in jdk7.
This resulted in modification to the readObject and writeObject methods,
and in particular, former serial fields were removed, resulting in
writeObject using PutField and readObject using defaultReadObject.
The writeObject invokes the putFields method of an ObjectOutputStream 
multiple times, and assumes
that it will receive the same PutField object instance for each 
invocation. The spec
doesn't endorse this behaviour - but that's what the implementation of 
ObjectOutputStream
provides. However in the case of RMI-IIOP, the OutputStreamHook, a 
subclass of ObjectOutputStream, returns a new instance for each
putFields invocation. Thus, the ConcurrentHashMap writeObject results in 
improper serialization in the context
of RMI-IIOP.
In the unmarshalling flow of ConcurrentHashMap, the readObject now uses 
defaultReadObject rather than GetField
In this call flow the IIOPInputStream attempts to ignore any primitive 
field, in the serialized data, that doesn't
correspond with a reflective field of the object. However, it leaves the 
primitive in the stream.
Thus, in the case of ConcurrentHashMap, which has serialized two 
integers and
an array of Segments (subclass of ReentrantLock), this results in erroneous
deserialization, with a misinterpretation of a value tag in the stream 
as an array length
and an attempt to allocate a very large array ensues, with an exception 
being thrown.

The OutputStreamHook now returns the same instance of PutField allocated 
for each separate call of putFields method.
This highlights a need to tighten up and disambiguate the 
OutputObjectStream spec for putFields.

IIOPInputStream now removes the primitive values from the stream.

regards
Mark