7044285: VM crashes in server app

[Yasumasa Suenaga]

Hi,

Our customer's system was also crashed in the same case.
I check core image, and I suspect overflow of "pDst" in "Java_sun_java2d_loops_MaskFill_MaskFill()"

In order to fix this problem, I made a patch for typecasting "ptrdiff_t" in PtrCoord macro.

Please merge this patch if you don't fix this problem yet.
("test.c" is not a patch. It is minimal sample of this overflow problem.)


from hs_err log:
----------------------------
#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00002aabcb644177, pid=27759, tid=1142659392
#
# Java VM: OpenJDK 64-Bit Server VM (1.6.0-b09 mixed mode linux-amd64)
# Problematic frame:
# C  [libawt.so+0x63177]  IntArgbSrcOverMaskFill+0x127
#
# If you would like to submit a bug report, please visit:
#   http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

                    :
                    :

OS:Red Hat Enterprise Linux Server release 5.4 (Tikanga)

uname:Linux 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE infinity, NPROC infinity, NOFILE 65536, AS infinity
load average:1.04 0.56 0.41

CPU:total 4 (1 cores per cpu, 1 threads per core) family 6 model 10 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 5830108k(39684k free), swap 4192956k(4065544k free)

vm_info: OpenJDK 64-Bit Server VM (1.6.0-b09) for linux-amd64 JRE (1.6.0-b09), built on Aug  5 2009 11:16:51 by "mockbuild" with gcc 4.1.2 20080704 (Red Hat 4.1.2-44)

time: Thu Jun  2 21:04:51 2011
elapsed time: 517630 seconds
----------------------------

from core image:
----------------------------
[root at RHEL5-4 T2011060009]# gdb java core.27759

        :
        :

(gdb) f 7
#7  0x00002aabcb61cd3d in Java_sun_java2d_loops_MaskFill_MaskFill (env=0x2aabcc36f598,
    self=<value optimized out>, sg2d=0x441b70c8, sData=<value optimized out>,
    comp=<value optimized out>, x=50, y=26188, w=32, h=32, maskArray=0x441b7120,
    maskoff=0, maskscan=32) at ../../../src/share/native/sun/java2d/loops/MaskFill.c:85
85      ../../../src/share/native/sun/java2d/loops/MaskFill.c: No such file or directory.
        in ../../../src/share/native/sun/java2d/loops/MaskFill.c
(gdb) p pDst
$1 = (void *) 0x2aaa8aaea6e0
(gdb) p rasInfo
$2 = {bounds = {x1 = 50, y1 = 26188, x2 = 82, y2 = 26220}, rasBase = 0x2aab0a4fc718,
  pixelBitOffset = 0, pixelStride = 4, scanStride = 82240, lutSize = 0, lutBase = 0x0,
  invColorTable = 0x0, redErrTable = 0x0, grnErrTable = 0x0, bluErrTable = 0x0,
  invGrayTable = 0x2aabb15d4d68, priv = {align = 0x3,
    data = "\003\000\000\000\000\000\000\000\030ヌO\nォ*", '\0' <repeats 18 times>, "@\000\000\000\000\000\000\000X\213P爼*\000\000\001", '\0' <repeats 14 times>}}
----------------------------

"pDst" is calculated in "MaskFill.c" as following:
----------------------------
           void *pDst = PtrCoord(rasInfo.rasBase,
                                  rasInfo.bounds.x1, rasInfo.pixelStride,
                                  rasInfo.bounds.y1, rasInfo.scanStride);
----------------------------

"PtrCoord" is defined in "GraphicsPrimitiveMgr.h":
----------------------------
#define PtrAddBytes(p, b)               ((void *) (((intptr_t) (p)) + (b)))
#define PtrCoord(p, x, xinc, y, yinc)   PtrAddBytes(p, (y)*(yinc) + (x)*(xinc))
----------------------------

In this case, "b" in PtrAddBytes macro is

  (rasInfo.bounds.y1 * rasInfo.scanStride) + (rasInfo.bounds.x1 * rasInfo.pixelStride)
   = (26188 * 82240) + (50 * 4)
   = 2153701320 ( > INT_MAX ( 2147483647 (0x7fffffff) ))

"b" sets to be -2141265976. So, "pDst" set to be as following:

pDst = rasInfo.bounds.rasBase - 2141265976
     = 0x2aaa8aaea6e0


pDst should set to be 0x2aab8aaea6e0,
however, it set to be 0x2aaa8aaea6e0.



Best regards,

Yasumasa
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test.c
Url: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20110606/dc23ad43/attachment.c 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 7044285.patch
Url: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20110606/dc23ad43/attachment.ksh