[OpenJDK 2D-Dev] [11] RFR for JDK-8201433: Fix potential crash in BufImg_SetupICM

[Alexey Ivanov]

Hi Sergey,

Thank you for your review.

Please take a look at the updated webrev:
http://cr.openjdk.java.net/~aivanov/8201433/jdk11/webrev.01/

On 12/04/2018 22:33, Sergey Bylokhov wrote:
> Hi, Alexey.
> Since the test requires 1g of memory, should we use this tag?:
>  * @requires os.maxMemory >= 1g
> Otherwise the test may fail on start if amount of memory is not 
> sufficient.

It makes sense to add it so that the test is skipped if there's not 
enough of memory.
However, I think the actual requirement is >= 2G as it needs 1G for Java 
heap and it needs native memory for images and for other purposes.

The test exhausts the available native memory to make malloc fail. The 
amount of used memory reaches close to 2G before it fails.


Regards,
Alexey

>
> On 12/04/2018 08:25, Alexey Ivanov wrote:
>> Hi,
>>
>> Please review the fix for jdk11:
>>
>> bug: https://bugs.openjdk.java.net/browse/JDK-8201433
>> webrev: http://cr.openjdk.java.net/~aivanov/8201433/jdk11/webrev.00/
>>
>>
>> When the JVM is run with limited amount of memory, initCubemap 
>> function called from BufImg_SetupICM can return NULL. It can lead to 
>> a crash. It usually happens when native memory is exhausted and 
>> malloc fails. In this state, JVM itself is not stable.
>>
>> Adding NULL-check can prevent the crash in this particular place. 
>> However, if the app continues, since native memory is exhausted and 
>> malloc fails, JVM can still crash in another place.
>>
>>
>> Thank you in advance.
>>
>> Regards,
>> Alexey