[OpenJDK 2D-Dev] [9] RFR JDK-6842011: StackOverflowError printing landscape with scale and transform

[prasanta sadhukhan]

Hi All,

Please review a fix for jdk9 whereby it is seen that
A StackOverflowError occurs when printing in landscape orientation with 
a scaled and transformed graphics object.
  at sun.print.PSPrinterJob.prepDrawing(PSPrinterJob.java:1610)
     at sun.print.PSPrinterJob.beginPath(PSPrinterJob.java:1319)
     at sun.print.PSPrinterJob.convertToPSPath(PSPrinterJob.java:1793)
     at sun.print.PSPrinterJob$GState.emitPSClip(PSPrinterJob.java:1718)
     at sun.print.PSPrinterJob.prepDrawing(PSPrinterJob.java:1625)

  at sun.print.PSPrinterJob.beginPath(PSPrinterJob.java:1319)
     at sun.print.PSPrinterJob.convertToPSPath(PSPrinterJob.java:1793)
     at sun.print.PSPrinterJob$GState.emitPSClip(PSPrinterJob.java:1718)
     at sun.print.PSPrinterJob.prepDrawing(PSPrinterJob.java:1625)

Bug: https://bugs.openjdk.java.net/browse/JDK-6842011
webrev: http://cr.openjdk.java.net/~psadhukhan/6842011/webrev.00/

StackOverflowError is occuring because the scalex, scaley for landscape 
orientation was 0 so when the testcase tries to scale with these scale 
factors
using g2d.scale( 1 / scalex, 1 / scaley );
it creates a AffineTransform of NaN transformation. Now, In linux, when 
the PS print drawing information is being prepared, it calls 
prepDrawing() where it checks
getGState().mTransform.equals(mLastTransform) and since NaN values 
cannot be compared it results in "false", causing erroneous "grestore" 
postscript command to be issued and remove a GState from the stack so 
isOuterGState() becomes true which causes emitPSClip() to be called 
which calls
prepDrawing() again via convertToPSPath() , beginPath() and since 
isOuterState() returns true due to transform not being equal it again 
calls emitPSClip() causing a recursion.

The fix was to check if transform is NaN and do not fill the devicePath 
if it is so, so that erroeous drawing is not done.
So, it will print out a blank page.

In windows, the testcase prints out a blank page. In mac, the testcase 
prints a 2x2 rectangle.

Regards
Prasanta