<AWT Dev> 7044285: VM crashes in server app

[Denis S. Fokin]

Hi David,

it seems like a java 2d issue.

Thank you,
Denis.

On 06/06/2011 04:08 PM, David Holmes wrote:
> This isn't a hotspot issue but an AWT (or rather java2d) issue. I've
> cc'ed the AWT folk and bcc'd hotspot.
>
> The incident report is mis-filed and I'll report that.
>
> David Holmes
>
> Yasumasa Suenaga said the following on 06/06/11 21:24:
>> Hi,
>>
>> Our customer's system was also crashed in the same case.
>> I check core image, and I suspect overflow of "pDst" in "Java_sun_java2d_loops_MaskFill_MaskFill()"
>>
>> In order to fix this problem, I made a patch for typecasting "ptrdiff_t" in PtrCoord macro.
>>
>> Please merge this patch if you don't fix this problem yet.
>> ("test.c" is not a patch. It is minimal sample of this overflow problem.)
>>
>>
>> from hs_err log:
>> ----------------------------
>> #
>> # An unexpected error has been detected by Java Runtime Environment:
>> #
>> #  SIGSEGV (0xb) at pc=0x00002aabcb644177, pid=27759, tid=1142659392
>> #
>> # Java VM: OpenJDK 64-Bit Server VM (1.6.0-b09 mixed mode linux-amd64)
>> # Problematic frame:
>> # C  [libawt.so+0x63177]  IntArgbSrcOverMaskFill+0x127
>> #
>> # If you would like to submit a bug report, please visit:
>> #   http://icedtea.classpath.org/bugzilla
>> # The crash happened outside the Java Virtual Machine in native code.
>> # See problematic frame for where to report the bug.
>>
>>                     :
>>                     :
>>
>> OS:Red Hat Enterprise Linux Server release 5.4 (Tikanga)
>>
>> uname:Linux 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64
>> libc:glibc 2.5 NPTL 2.5
>> rlimit: STACK 10240k, CORE infinity, NPROC infinity, NOFILE 65536, AS infinity
>> load average:1.04 0.56 0.41
>>
>> CPU:total 4 (1 cores per cpu, 1 threads per core) family 6 model 10 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
>>
>> Memory: 4k page, physical 5830108k(39684k free), swap 4192956k(4065544k free)
>>
>> vm_info: OpenJDK 64-Bit Server VM (1.6.0-b09) for linux-amd64 JRE (1.6.0-b09), built on Aug  5 2009 11:16:51 by "mockbuild" with gcc 4.1.2 20080704 (Red Hat 4.1.2-44)
>>
>> time: Thu Jun  2 21:04:51 2011
>> elapsed time: 517630 seconds
>> ----------------------------
>>
>> from core image:
>> ----------------------------
>> [root at RHEL5-4 T2011060009]# gdb java core.27759
>>
>>         :
>>         :
>>
>> (gdb) f 7
>> #7  0x00002aabcb61cd3d in Java_sun_java2d_loops_MaskFill_MaskFill (env=0x2aabcc36f598,
>>     self=<value optimized out>, sg2d=0x441b70c8, sData=<value optimized out>,
>>     comp=<value optimized out>, x=50, y=26188, w=32, h=32, maskArray=0x441b7120,
>>     maskoff=0, maskscan=32) at ../../../src/share/native/sun/java2d/loops/MaskFill.c:85
>> 85      ../../../src/share/native/sun/java2d/loops/MaskFill.c: No such file or directory.
>>         in ../../../src/share/native/sun/java2d/loops/MaskFill.c
>> (gdb) p pDst
>> $1 = (void *) 0x2aaa8aaea6e0
>> (gdb) p rasInfo
>> $2 = {bounds = {x1 = 50, y1 = 26188, x2 = 82, y2 = 26220}, rasBase = 0x2aab0a4fc718,
>>   pixelBitOffset = 0, pixelStride = 4, scanStride = 82240, lutSize = 0, lutBase = 0x0,
>>   invColorTable = 0x0, redErrTable = 0x0, grnErrTable = 0x0, bluErrTable = 0x0,
>>   invGrayTable = 0x2aabb15d4d68, priv = {align = 0x3,
>>     data = "\003\000\000\000\000\000\000\000\030ヌO\nォ*", '\0' <repeats 18 times>, "@\000\000\000\000\000\000\000X\213P爼*\000\000\001", '\0' <repeats 14 times>}}
>> ----------------------------
>>
>> "pDst" is calculated in "MaskFill.c" as following:
>> ----------------------------
>>            void *pDst = PtrCoord(rasInfo.rasBase,
>>                                   rasInfo.bounds.x1, rasInfo.pixelStride,
>>                                   rasInfo.bounds.y1, rasInfo.scanStride);
>> ----------------------------
>>
>> "PtrCoord" is defined in "GraphicsPrimitiveMgr.h":
>> ----------------------------
>> #define PtrAddBytes(p, b)               ((void *) (((intptr_t) (p)) + (b)))
>> #define PtrCoord(p, x, xinc, y, yinc)   PtrAddBytes(p, (y)*(yinc) + (x)*(xinc))
>> ----------------------------
>>
>> In this case, "b" in PtrAddBytes macro is
>>
>>   (rasInfo.bounds.y1 * rasInfo.scanStride) + (rasInfo.bounds.x1 * rasInfo.pixelStride)
>>    = (26188 * 82240) + (50 * 4)
>>    = 2153701320 ( > INT_MAX ( 2147483647 (0x7fffffff) ))
>>
>> "b" sets to be -2141265976. So, "pDst" set to be as following:
>>
>> pDst = rasInfo.bounds.rasBase - 2141265976
>>      = 0x2aaa8aaea6e0
>>
>>
>> pDst should set to be 0x2aab8aaea6e0,
>> however, it set to be 0x2aaa8aaea6e0.
>>
>>
>>
>> Best regards,
>>
>> Yasumasa
>>