RFR: 10: 8160638: solaris JVM unable to allocate more than 2GB of direct byte buffers when max heap is <= 2GB

[Vladimir Kempik]

Hello

Please review this fix for bug JDK-8160638 
<https://bugs.openjdk.java.net/browse/JDK-8160638>

The issue is with solaris only.

When java mmaps heap (with compressed Oops enabled), mmaped heap become 
upper limit for any native mallocs.

So when heap is starting at 2 gb, the maximum we can malloc is 2gb.

Native malloc is used by Direct Memory Buffers, so even with 
-XX:MaxDirectMemorySize=100g we are still limited with less than 2 gb of 
memory for Direct Memory Buffers.

The fix moves HeapBaseMinAddress to upper space when it's needed for 
MaxDirectMemorySize to operate properly, leaving about 1 gb of native 
memory for java's needs.

Testing: jprt, included testcase.

Webrev - http://cr.openjdk.java.net/~vkempik/8160638/webrev.00/

Bug - https://bugs.openjdk.java.net/browse/JDK-8160638

Thanks, Vladimir