Mismatch between JDK and JVM re largest byte array that VM can allocate?

[Srinivas Ramakrishna]

It's causing an issue -- which can be worked around in our code -- which is how I stumbled upon this. My thinking was that perhaps the jvm limit could be communicated to the libs via a suitable jvm method.  Or better still the unnecessary limitation could be removed for the 64 bit jvm and everyone's happy.

Max sized byte arrays are not that uncommon in certain kinds of apps using large heaps.

The unfortunate part is that the workaround might require one to stay well away from the actual limit -- to the extent of less than half of max-int minus  header size.

So, definitely not a show-stopper, but would be nice to fix-- and perhaps easy to in the 64 bit case which is where it matters anyway.

-- Ramki

ysr1729

On Mar 11, 2013, at 18:25, David Holmes <david.holmes at oracle.com> wrote:

> Hi Ramki,
> 
> The maximum array size is a VM limitation based on the internal implementation, so I don't think the JDK code should be aware of this limitation.
> 
> At least with the present code the request for a size of Integer.MAX_VALUE will fail immediately, rather than spending half an hour failing to allocate an insanely large array ;-)
> 
> If huge arrays eventually make their way into Java we will have to address this, but otherwise it seems pretty low priority to me.
> 
> Is this actually causing an issue or is it just an observation?
> 
> Cheers,
> David
> 
> On 12/03/2013 9:11 AM, Srinivas Ramakrishna wrote:
>> I am looking at code in (for example) ByteArrayOutputStream.java :-
>> 
>>        96     /**
>> 
>>        97      * Increases the capacity to ensure that it can hold at least the
>> 
>>        98      * number of elements specified by the minimum capacity argument.
>> 
>>        99      *
>> 
>>       100      * @param minCapacity the desired minimum capacity
>> 
>>       101      */
>> 
>>       102     private void grow(int minCapacity) {
>> 
>>       103         // overflow-conscious code
>> 
>>       104         int oldCapacity = buf.length;
>> 
>>       105         int newCapacity = oldCapacity << 1;
>> 
>>       106         if (newCapacity - minCapacity < 0)
>> 
>>       107             newCapacity = minCapacity;
>> 
>>       108         if (newCapacity < 0) {
>> 
>>       109             if (minCapacity < 0) // overflow
>> 
>>       110                 throw new OutOfMemoryError();
>> 
>>       111             newCapacity = Integer.MAX_VALUE;
>> 
>>       112         }
>> 
>>       113         buf = Arrays.copyOf(buf, newCapacity);
>> 
>>       114     }
>> 
>> 
>> This can result in a request for an array of size Integer.MAX_VALUE
>> (because of line 111 above), see below:-
>> 
>>      2874     /**
>> 
>>      2875      * Copies the specified array, truncating or padding
>> with zeros (if necessary)
>> 
>>      2876      * so the copy has the specified length.  For all indices that are
>> 
>>      2877      * valid in both the original array and the copy, the
>> two arrays will
>> 
>>      2878      * contain identical values.  For any indices that are
>> valid in the
>> 
>>      2879      * copy but not the original, the copy will contain
>> <tt>(byte)0</tt>.
>> 
>>      2880      * Such indices will exist if and only if the specified length
>> 
>>      2881      * is greater than that of the original array.
>> 
>>      2882      *
>> 
>>      2883      * @param original the array to be copied
>> 
>>      2884      * @param newLength the length of the copy to be returned
>> 
>>      2885      * @return a copy of the original array, truncated or
>> padded with zeros
>> 
>>      2886      *     to obtain the specified length
>> 
>>      2887      * @throws NegativeArraySizeException if
>> <tt>newLength</tt> is negative
>> 
>>      2888      * @throws NullPointerException if <tt>original</tt> is null
>> 
>>      2889      * @since 1.6
>> 
>>      2890      */
>> 
>>      2891     public static byte[] copyOf(byte[] original, int newLength) {
>> 
>>      2892         byte[] copy = new byte[newLength];
>> 
>>      2893         System.arraycopy(original, 0, copy, 0,
>> 
>>      2894                          Math.min(original.length, newLength));
>> 
>>      2895         return copy;
>> 
>>      2896     }
>> 
>> 
>> So the call at line 2892 can cause a request for an array with
>> Integer.MAX_VALUE entries, yet the JVM doesn't give you arrays that
>> large; witness this code in arrayOop.cpp where this will flatten out:-
>> 
>>   // Return the maximum length of an array of BasicType.  The length can passed
>>   // to typeArrayOop::object_size(scale, length, header_size) without causing an
>>   // overflow. We also need to make sure that this will not overflow a size_t on
>>   // 32 bit platforms when we convert it to a byte size.
>>   static int32_t max_array_length(BasicType type) {
>>     assert(type >= 0 && type < T_CONFLICT, "wrong type");
>>     assert(type2aelembytes(type) != 0, "wrong type");
>> 
>>     const size_t max_element_words_per_size_t =
>>       align_size_down((SIZE_MAX/HeapWordSize - header_size(type)),
>> MinObjAlignment);
>>     const size_t max_elements_per_size_t =
>>       HeapWordSize * max_element_words_per_size_t / type2aelembytes(type);
>>     if ((size_t)max_jint < max_elements_per_size_t) {
>>       // It should be ok to return max_jint here, but parts of the code
>>       // (CollectedHeap, Klass::oop_oop_iterate(), and more) uses an int for
>>       // passing around the size (in words) of an object. So, we need to avoid
>>       // overflowing an int when we add the header. See CRs 4718400 and 7110613.
>>       return align_size_down(max_jint - header_size(type), MinObjAlignment);
>>     }
>>     return (int32_t)max_elements_per_size_t;
>>   }
>> 
>> 
>> Is there any plan to fix this mismatch ? May be it's as simple(!) as
>> checking the GC code to make sure it doesn't traffic in int's and
>> fix the code above to return max_jint for the byte array case as well?
>> I have a vague recollection of a bug id already for this, but
>> it's been a while....
>> 
>> Thanks!
>> -- ramki
>>