[OpenJDK 2D-Dev] [9] RFR JDK-6842011: StackOverflowError printing landscape with scale and transform

[prasanta sadhukhan]

Hi Phil,

When we call print() it calls RasterPrinterJob#printPage() which sets 
peekGraphics.transform([4.16,0,0][0,4.16,0]) as obtained from 
xscale=4.16 [getXRes()=300 / 72.0] ,yscale=4.16
It calls SunGraphics2D.transform which was identity transform [1.0, 0.0, 
0.0] [0.0, 1.0, 0.0] calls transform.concatenate(peekgraphicsTx) and 
stores as ([4.16,0,0][0,4.16,0])

Then RasterPrinterJob#printPage() again calls peekGraphics.transform(new 
AffineTransform(page.getMatrix()));
where page.getMatrix() returns 0.0, -1.0, 1.0, 0.0, 0.0, 841.88 and 
peekGraphics transform now becomes [0.0, 1.0, 0.0] [-1.0, 0.0, 841.88]
which calls SunGraphics2D#transform() where it again does 
transform.concatenate(peekgraphicsTx)

so the transform becomes [m00=0, m01=4.16, m02=0][m10=-4.16, m11=0, 
m12=3507.873]
Now scaleX obtains value from g2d.getTransform().getScaleX() which 
returns SunGraphics2D stored transform.m00 which is 0 and scaleY is 
m11=0 so scaleX,scaleY becomes 0.

Regards
Prasanta
On 5/19/2016 4:03 AM, Phil Race wrote:
> It sounds like scalex & scaley are 0 and are then used in calculations 
> which
> results in the NaN ? So why are they zero to begin with ?
>
> -phil.
>
> On 5/16/2016 3:32 AM, prasanta sadhukhan wrote:
>> Hi All,
>>
>> Please review a fix for jdk9 whereby it is seen that
>> A StackOverflowError occurs when printing in landscape orientation 
>> with a scaled and transformed graphics object.
>>  at sun.print.PSPrinterJob.prepDrawing(PSPrinterJob.java:1610)
>>     at sun.print.PSPrinterJob.beginPath(PSPrinterJob.java:1319)
>>     at sun.print.PSPrinterJob.convertToPSPath(PSPrinterJob.java:1793)
>>     at sun.print.PSPrinterJob$GState.emitPSClip(PSPrinterJob.java:1718)
>>     at sun.print.PSPrinterJob.prepDrawing(PSPrinterJob.java:1625)
>>
>>  at sun.print.PSPrinterJob.beginPath(PSPrinterJob.java:1319)
>>     at sun.print.PSPrinterJob.convertToPSPath(PSPrinterJob.java:1793)
>>     at sun.print.PSPrinterJob$GState.emitPSClip(PSPrinterJob.java:1718)
>>     at sun.print.PSPrinterJob.prepDrawing(PSPrinterJob.java:1625)
>>
>> Bug: https://bugs.openjdk.java.net/browse/JDK-6842011
>> webrev: http://cr.openjdk.java.net/~psadhukhan/6842011/webrev.00/
>>
>> StackOverflowError is occuring because the scalex, scaley for 
>> landscape orientation was 0 so when the testcase tries to scale with 
>> these scale factors
>> using g2d.scale( 1 / scalex, 1 / scaley );
>> it creates a AffineTransform of NaN transformation. Now, In linux, 
>> when the PS print drawing information is being prepared, it calls 
>> prepDrawing() where it checks
>> getGState().mTransform.equals(mLastTransform) and since NaN values 
>> cannot be compared it results in "false", causing erroneous 
>> "grestore" postscript command to be issued and remove a GState from 
>> the stack so isOuterGState() becomes true which causes emitPSClip() 
>> to be called which calls
>> prepDrawing() again via convertToPSPath() , beginPath() and since 
>> isOuterState() returns true due to transform not being equal it again 
>> calls emitPSClip() causing a recursion.
>>
>> The fix was to check if transform is NaN and do not fill the 
>> devicePath if it is so, so that erroeous drawing is not done.
>> So, it will print out a blank page.
>>
>> In windows, the testcase prints out a blank page. In mac, the 
>> testcase prints a 2x2 rectangle.
>>
>> Regards
>> Prasanta
>
>