[OpenJDK 2D-Dev] [9]Fix for JDK-8158356 : SIGSEGV when attempting to rotate BufferedImage using AffineTransform by NaN degrees

[Ajit Ghaisas]

Hi,

   The root cause of the crash is - NaN is converted to an integer and used as array index in mlib_ImageScanPoly.c.
   The native method previously did not check the validity of the input double argument. Now, I have added a check for finite double values.
   
   If NaN or INF arguments are present in affine transform, the changes done in native code result in ImagingOpException ("Unable to transform src image") in AffineTransformOp.filter() methods.
   The constructors in AffineTransformOp.java are left unchanged. 
   Please note that, this fix fixes the crash. The behavioral change in AffineTransformOp for such inputs will be fixed under JDK-8164729.

   Please review the updated webrev.
   http://cr.openjdk.java.net/~aghaisas/8158356/webrev.01/

Regards,
Ajit

-----Original Message-----
From: Philip Race 
Sent: Thursday, August 11, 2016 3:22 AM
To: Jim Graham
Cc: Ajit Ghaisas; 2d-dev
Subject: Re: [OpenJDK 2D-Dev] [9]Fix for JDK-8158356 : SIGSEGV when attempting to rotate BufferedImage using AffineTransform by NaN degrees

Agreed, I had previously asked for that too (off-line).
ie. root cause why a NaN would cause a crash ..

-phil.

On 8/10/16, 2:47 PM, Jim Graham wrote:
> This does address the specific test case directly, but I'd be happier 
> if we dug down and figured out where it went wrong in trying to 
> transform the image and put in a fix that addressed the root problem 
> whether it comes from the inputs being NaN or from some other similar 
> condition that could also trigger the same poorly written transform 
> code...
>
>             ...jim
>
> On 8/10/16 2:22 PM, Phil Race wrote:
>> 1) The spec for the constructors needs to be updated to include this 
>> reason for throwing ImagingOpException. A CCC request will be needed.
>>
>> 2) The C usage of "isnan()" may be problematic in some compilation 
>> environments.
>> For example I believe this will not compile with VS2010, and many 
>> folks still use that.
>> Instead you could use matrix[j] != matrix[j] as the two values should 
>> not compare equal if it is NaN.
>>
>> -phil.
>>
>> On 08/10/2016 04:15 AM, Ajit Ghaisas wrote:
>>> Hi,
>>>
>>>      Bug : https://bugs.openjdk.java.net/browse/JDK-8158356
>>>
>>>      Issue : AffineTransform using NaN value as input parameter 
>>> results in SIGSEGV.
>>>
>>>      Fix : Transformation matrix is checked for NaN values in 
>>> AffineTransformOp.validateTransform().
>>>               Also, at native level a separate check is made to 
>>> return error in case of NaN values.
>>>
>>>      Webrev : 
>>> http://cr.openjdk.java.net/~aghaisas/8158356/webrev.00/
>>>
>>>      Request you to review.
>>>
>>> Regards,
>>> Ajit
>>>
>>