[PATCH] RFC: XML encoder can cause a StackOverflowError (PR295)

[Omair Majid]

Andrew Haley wrote:
> Omair Majid wrote:
> 
>> IcedTea6 and apparently OpenJDK6, IcedTea, OpenJDK and the Xalan-J2 all
>> have a bug in their xml encoder, which can result in a
>> StackOverflowError [1].
>>
>> The test case which tries to encode character 4096 to 321 easily
>> reproduces the issue.  The issue appears to be this: When finding the
>> encoding for (char) 4096, an EncodingImpl object is created which
>> explicitly manages characters 4096 to 4223. So far so good. But when the
>> transformer tries to find the encoding for (char) 4095, a new
>> EncodingImpl delegate object is created which manages the values 4095 to
>> 4222. Effectively, this object only manages the value 4095 (since the
>> parent already manages 4096 to 4222). To find the value for 4094, a new
>> delegate is created. Do this a few more times, and you have one stack
>> frame for each value. Trying to encode characters from 0xffff to 0x0000
>> will result in thousands of stack frames.
>>
>> The proposed patch makes sure that each delegate manages 128 values
>> which dont overlap with any other delegate. The patch brings down the
>> maximum number of delegates in the chain to (0xffff + 1)/RANGE = 512
>> (also the max number of stack frames that can be used by EndcodingImpl).
> 
> Thanks.  This is OK for IcedTea.

Committed as
2009-02-26  Omair Majid  <omajid at redhat.com>

     * patches/icedtea-xml-encodinginfo.patch: New file. Fix possible
     StackOverflowError.
     * Makefile.am (ICEDTEA_PATCHES): Apply the above.
     * HACKING: Document the above.

> Please create an upstream bug for this, attaching the test case and your
> patch.

Filed at http://bugs.openjdk.java.net/show_bug.cgi?id=100017

> Andrew.

Cheers,
Omair