7044285: VM crashes in server app

[Paul Hohensee]

This doesn't look like a jvm problem, rather it's a problem in the graphics
library. At least, that's where the code is. I'm not sure who's responsible
for java2d these days, so I've cc'ed Andrey Pikalev (swing/awt manager)
and Rich Bair (client java architect).

Paul

On 6/6/11 7:24 AM, Yasumasa Suenaga wrote:
> Hi,
>
> Our customer's system was also crashed in the same case.
> I check core image, and I suspect overflow of "pDst" in "Java_sun_java2d_loops_MaskFill_MaskFill()"
>
> In order to fix this problem, I made a patch for typecasting "ptrdiff_t" in PtrCoord macro.
>
> Please merge this patch if you don't fix this problem yet.
> ("test.c" is not a patch. It is minimal sample of this overflow problem.)
>
>
> from hs_err log:
> ----------------------------
> #
> # An unexpected error has been detected by Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00002aabcb644177, pid=27759, tid=1142659392
> #
> # Java VM: OpenJDK 64-Bit Server VM (1.6.0-b09 mixed mode linux-amd64)
> # Problematic frame:
> # C  [libawt.so+0x63177]  IntArgbSrcOverMaskFill+0x127
> #
> # If you would like to submit a bug report, please visit:
> #   http://icedtea.classpath.org/bugzilla
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
>
>                     :
>                     :
>
> OS:Red Hat Enterprise Linux Server release 5.4 (Tikanga)
>
> uname:Linux 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64
> libc:glibc 2.5 NPTL 2.5
> rlimit: STACK 10240k, CORE infinity, NPROC infinity, NOFILE 65536, AS infinity
> load average:1.04 0.56 0.41
>
> CPU:total 4 (1 cores per cpu, 1 threads per core) family 6 model 10 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
>
> Memory: 4k page, physical 5830108k(39684k free), swap 4192956k(4065544k free)
>
> vm_info: OpenJDK 64-Bit Server VM (1.6.0-b09) for linux-amd64 JRE (1.6.0-b09), built on Aug  5 2009 11:16:51 by "mockbuild" with gcc 4.1.2 20080704 (Red Hat 4.1.2-44)
>
> time: Thu Jun  2 21:04:51 2011
> elapsed time: 517630 seconds
> ----------------------------
>
> from core image:
> ----------------------------
> [root at RHEL5-4 T2011060009]# gdb java core.27759
>
>         :
>         :
>
> (gdb) f 7
> #7  0x00002aabcb61cd3d in Java_sun_java2d_loops_MaskFill_MaskFill (env=0x2aabcc36f598,
>     self=<value optimized out>, sg2d=0x441b70c8, sData=<value optimized out>,
>     comp=<value optimized out>, x=50, y=26188, w=32, h=32, maskArray=0x441b7120,
>     maskoff=0, maskscan=32) at ../../../src/share/native/sun/java2d/loops/MaskFill.c:85
> 85      ../../../src/share/native/sun/java2d/loops/MaskFill.c: No such file or directory.
>         in ../../../src/share/native/sun/java2d/loops/MaskFill.c
> (gdb) p pDst
> $1 = (void *) 0x2aaa8aaea6e0
> (gdb) p rasInfo
> $2 = {bounds = {x1 = 50, y1 = 26188, x2 = 82, y2 = 26220}, rasBase = 0x2aab0a4fc718,
>   pixelBitOffset = 0, pixelStride = 4, scanStride = 82240, lutSize = 0, lutBase = 0x0,
>   invColorTable = 0x0, redErrTable = 0x0, grnErrTable = 0x0, bluErrTable = 0x0,
>   invGrayTable = 0x2aabb15d4d68, priv = {align = 0x3,
>     data = "\003\000\000\000\000\000\000\000\030ヌO\nォ*", '\0' <repeats 18 times>, "@\000\000\000\000\000\000\000X\213P爼*\000\000\001", '\0' <repeats 14 times>}}
> ----------------------------
>
> "pDst" is calculated in "MaskFill.c" as following:
> ----------------------------
>            void *pDst = PtrCoord(rasInfo.rasBase,
>                                   rasInfo.bounds.x1, rasInfo.pixelStride,
>                                   rasInfo.bounds.y1, rasInfo.scanStride);
> ----------------------------
>
> "PtrCoord" is defined in "GraphicsPrimitiveMgr.h":
> ----------------------------
> #define PtrAddBytes(p, b)               ((void *) (((intptr_t) (p)) + (b)))
> #define PtrCoord(p, x, xinc, y, yinc)   PtrAddBytes(p, (y)*(yinc) + (x)*(xinc))
> ----------------------------
>
> In this case, "b" in PtrAddBytes macro is
>
>   (rasInfo.bounds.y1 * rasInfo.scanStride) + (rasInfo.bounds.x1 * rasInfo.pixelStride)
>    = (26188 * 82240) + (50 * 4)
>    = 2153701320 ( > INT_MAX ( 2147483647 (0x7fffffff) ))
>
> "b" sets to be -2141265976. So, "pDst" set to be as following:
>
> pDst = rasInfo.bounds.rasBase - 2141265976
>      = 0x2aaa8aaea6e0
>
>
> pDst should set to be 0x2aab8aaea6e0,
> however, it set to be 0x2aaa8aaea6e0.
>
>
>
> Best regards,
>
> Yasumasa