Mismatch between JDK and JVM re largest byte array that VM can allocate?

[David Holmes]

Hi Ramki,

The maximum array size is a VM limitation based on the internal 
implementation, so I don't think the JDK code should be aware of this 
limitation.

At least with the present code the request for a size of 
Integer.MAX_VALUE will fail immediately, rather than spending half an 
hour failing to allocate an insanely large array ;-)

If huge arrays eventually make their way into Java we will have to 
address this, but otherwise it seems pretty low priority to me.

Is this actually causing an issue or is it just an observation?

Cheers,
David

On 12/03/2013 9:11 AM, Srinivas Ramakrishna wrote:
> I am looking at code in (for example) ByteArrayOutputStream.java :-
>
>         96     /**
>
>         97      * Increases the capacity to ensure that it can hold at least the
>
>         98      * number of elements specified by the minimum capacity argument.
>
>         99      *
>
>        100      * @param minCapacity the desired minimum capacity
>
>        101      */
>
>        102     private void grow(int minCapacity) {
>
>        103         // overflow-conscious code
>
>        104         int oldCapacity = buf.length;
>
>        105         int newCapacity = oldCapacity << 1;
>
>        106         if (newCapacity - minCapacity < 0)
>
>        107             newCapacity = minCapacity;
>
>        108         if (newCapacity < 0) {
>
>        109             if (minCapacity < 0) // overflow
>
>        110                 throw new OutOfMemoryError();
>
>        111             newCapacity = Integer.MAX_VALUE;
>
>        112         }
>
>        113         buf = Arrays.copyOf(buf, newCapacity);
>
>        114     }
>
>
> This can result in a request for an array of size Integer.MAX_VALUE
> (because of line 111 above), see below:-
>
>       2874     /**
>
>       2875      * Copies the specified array, truncating or padding
> with zeros (if necessary)
>
>       2876      * so the copy has the specified length.  For all indices that are
>
>       2877      * valid in both the original array and the copy, the
> two arrays will
>
>       2878      * contain identical values.  For any indices that are
> valid in the
>
>       2879      * copy but not the original, the copy will contain
> <tt>(byte)0</tt>.
>
>       2880      * Such indices will exist if and only if the specified length
>
>       2881      * is greater than that of the original array.
>
>       2882      *
>
>       2883      * @param original the array to be copied
>
>       2884      * @param newLength the length of the copy to be returned
>
>       2885      * @return a copy of the original array, truncated or
> padded with zeros
>
>       2886      *     to obtain the specified length
>
>       2887      * @throws NegativeArraySizeException if
> <tt>newLength</tt> is negative
>
>       2888      * @throws NullPointerException if <tt>original</tt> is null
>
>       2889      * @since 1.6
>
>       2890      */
>
>       2891     public static byte[] copyOf(byte[] original, int newLength) {
>
>       2892         byte[] copy = new byte[newLength];
>
>       2893         System.arraycopy(original, 0, copy, 0,
>
>       2894                          Math.min(original.length, newLength));
>
>       2895         return copy;
>
>       2896     }
>
>
> So the call at line 2892 can cause a request for an array with
> Integer.MAX_VALUE entries, yet the JVM doesn't give you arrays that
> large; witness this code in arrayOop.cpp where this will flatten out:-
>
>    // Return the maximum length of an array of BasicType.  The length can passed
>    // to typeArrayOop::object_size(scale, length, header_size) without causing an
>    // overflow. We also need to make sure that this will not overflow a size_t on
>    // 32 bit platforms when we convert it to a byte size.
>    static int32_t max_array_length(BasicType type) {
>      assert(type >= 0 && type < T_CONFLICT, "wrong type");
>      assert(type2aelembytes(type) != 0, "wrong type");
>
>      const size_t max_element_words_per_size_t =
>        align_size_down((SIZE_MAX/HeapWordSize - header_size(type)),
> MinObjAlignment);
>      const size_t max_elements_per_size_t =
>        HeapWordSize * max_element_words_per_size_t / type2aelembytes(type);
>      if ((size_t)max_jint < max_elements_per_size_t) {
>        // It should be ok to return max_jint here, but parts of the code
>        // (CollectedHeap, Klass::oop_oop_iterate(), and more) uses an int for
>        // passing around the size (in words) of an object. So, we need to avoid
>        // overflowing an int when we add the header. See CRs 4718400 and 7110613.
>        return align_size_down(max_jint - header_size(type), MinObjAlignment);
>      }
>      return (int32_t)max_elements_per_size_t;
>    }
>
>
> Is there any plan to fix this mismatch ? May be it's as simple(!) as
> checking the GC code to make sure it doesn't traffic in int's and
> fix the code above to return max_jint for the byte array case as well?
> I have a vague recollection of a bug id already for this, but
> it's been a while....
>
> Thanks!
> -- ramki
>