RFR (M): 8129419: heapDumper.cpp: assert(length_in_bytes > 0) failed: nothing to copy

[Andreas Eriksson]

Hi,

Please review this fix for dumping of long arrays, and general cleanup 
of types in heapDumper.cpp.

Problem:
At several places in heapDumper.cpp overflows could happen when dumping 
long arrays.
Also the hprof format uses an u4 as a record length field, but arrays 
can be longer than that (counted in bytes).

Fix:
Many types that were previously signed are changed to equivalent 
unsigned types and/or to a larger type to prevent overflow.
The bulk of the change though is the addition of 
calculate_array_max_length, which for a given array returns the number 
of elements we can dump. That length is then used to truncate arrays 
that are too long.
Whenever an array is truncated a warning is printed:
Java HotSpot(TM) 64-Bit Server VM warning: cannot dump array of type 
object[] with length 1,073,741,823; truncating to length 536,870,908

Much of the rest of the change is moving functions needed by 
calculate_array_max_length to the DumpWriter or DumperSupport class so 
that they can be accessed.

Added a test that relies on the hprof parser, which also had a couple of 
overflow problems (top repo changes).
I've also run this change against a couple of other tests, but they are 
problematic in JPRT because they are using large heaps and lots of disk.

Bug:
8129419: heapDumper.cpp: assert(length_in_bytes > 0) failed: nothing to copy
https://bugs.openjdk.java.net/browse/JDK-8129419

Also fixed in this change is the problems seen in these two bugs:
8133317: Integer overflow in heapDumper.cpp leads to corrupt HPROF dumps
https://bugs.openjdk.java.net/browse/JDK-8133317

8144732: VM_HeapDumper hits assert with bad dump_len
https://bugs.openjdk.java.net/browse/JDK-8144732

Webrev:
Top repo: http://cr.openjdk.java.net/~aeriksso/8129419/webrev.00/jdk9-hs-rt/
Hotspot: http://cr.openjdk.java.net/~aeriksso/8129419/webrev.00/hotspot/

Regards,
Andreas