[OpenJDK 2D-Dev] [9]Fix for JDK-8158356 : SIGSEGV when attempting to rotate BufferedImage using AffineTransform by NaN degrees

[Jim Graham]

+1

		...jim

On 8/24/16 4:41 AM, Ajit Ghaisas wrote:
> Hi,
>
>    The root cause of the crash is - NaN is converted to an integer and used as array index in mlib_ImageScanPoly.c.
>    The native method previously did not check the validity of the input double argument. Now, I have added a check for finite double values.
>
>    If NaN or INF arguments are present in affine transform, the changes done in native code result in ImagingOpException ("Unable to transform src image") in AffineTransformOp.filter() methods.
>    The constructors in AffineTransformOp.java are left unchanged.
>    Please note that, this fix fixes the crash. The behavioral change in AffineTransformOp for such inputs will be fixed under JDK-8164729.
>
>    Please review the updated webrev.
>    http://cr.openjdk.java.net/~aghaisas/8158356/webrev.01/
>
> Regards,
> Ajit
>
> -----Original Message-----
> From: Philip Race
> Sent: Thursday, August 11, 2016 3:22 AM
> To: Jim Graham
> Cc: Ajit Ghaisas; 2d-dev
> Subject: Re: [OpenJDK 2D-Dev] [9]Fix for JDK-8158356 : SIGSEGV when attempting to rotate BufferedImage using AffineTransform by NaN degrees
>
> Agreed, I had previously asked for that too (off-line).
> ie. root cause why a NaN would cause a crash ..
>
> -phil.
>
> On 8/10/16, 2:47 PM, Jim Graham wrote:
>> This does address the specific test case directly, but I'd be happier
>> if we dug down and figured out where it went wrong in trying to
>> transform the image and put in a fix that addressed the root problem
>> whether it comes from the inputs being NaN or from some other similar
>> condition that could also trigger the same poorly written transform
>> code...
>>
>>             ...jim
>>
>> On 8/10/16 2:22 PM, Phil Race wrote:
>>> 1) The spec for the constructors needs to be updated to include this
>>> reason for throwing ImagingOpException. A CCC request will be needed.
>>>
>>> 2) The C usage of "isnan()" may be problematic in some compilation
>>> environments.
>>> For example I believe this will not compile with VS2010, and many
>>> folks still use that.
>>> Instead you could use matrix[j] != matrix[j] as the two values should
>>> not compare equal if it is NaN.
>>>
>>> -phil.
>>>
>>> On 08/10/2016 04:15 AM, Ajit Ghaisas wrote:
>>>> Hi,
>>>>
>>>>      Bug : https://bugs.openjdk.java.net/browse/JDK-8158356
>>>>
>>>>      Issue : AffineTransform using NaN value as input parameter
>>>> results in SIGSEGV.
>>>>
>>>>      Fix : Transformation matrix is checked for NaN values in
>>>> AffineTransformOp.validateTransform().
>>>>               Also, at native level a separate check is made to
>>>> return error in case of NaN values.
>>>>
>>>>      Webrev :
>>>> http://cr.openjdk.java.net/~aghaisas/8158356/webrev.00/
>>>>
>>>>      Request you to review.
>>>>
>>>> Regards,
>>>> Ajit
>>>>
>>>