[8u40] RFR 8055289: Internal Error: mallocTracker.cpp:146 fatal error: Should not use malloc for big memory block, use virtual memory instead

[Coleen Phillimore]

Please approve backport for this bug fix for NMT.  The patch applied 
with fuzz of 3 lines.

Summary: Return NULL if memory allocation size is bigger than 
MAX_MALLOC_SIZE when NMT is on

open webrev at http://cr.openjdk.java.net/~coleenp/8055289_8u40/
bug link https://bugs.openjdk.java.net/browse/JDK-8055289

Thanks,
Coleen