[OpenJDK 2D-Dev] <AWT Dev> [8] Review request for 8005607: Recursion in J2DXErrHandler() Causes a Stack Overflow on Linux

[Jim Graham]

I think I'd rather see some way to prevent double-adding the handler in 
the first place as well.  Since it is only ever used on errors I also 
think it is OK to set it once and leave it there forever...

			...jim

On 1/9/13 8:08 AM, Anthony Petrov wrote:
> Hi Anton et al.,
>
> If I read the description of the bug correctly, specifically this part:
>
>> The problem occurs, if another thread (for example, GTK thread) is
>> doing the same sort of thing concurrently. This can lead to the
>> following situation.
>>  JVM thread: Sets J2DXErrHandler(), saves ANY_PREVIOUS_HANDLER as
>> previous      GTK thread: Sets some GTK_HANDLER, saves
>> J2DXErrHandler() as previous  JVM thread: Restores
>> ANY_PREVIOUS_HANDLER      GTK thread: Restores J2DXErrHandler()  JVM
>> thread: Sets J2DXErrHandler(), saves J2DXErrHandler() as previous
>
> It is obvious that at this final step 2D is in an inconsistent state. We
> don't expect to replace our own error handler (and it shouldn't have
> been there in the first place).
>
> I realize that the fix you propose works around this problem. But this
> doesn't look like an ideal solution to me.
>
> BTW, IIRC, in JDK7 (and 6?) we decided to set the actual X11 error
> handler only once and never replace it. All the rest of the
> push_handler/pop_handler logic is now located in Java code (see
> XToolkit.SAVED_ERROR_HANDLER() and the surrounding logic). I believe
> that we should somehow share this machinery with the 2D code to avoid
> this sort of problems. Though I'm not sure if this will eliminate this
> exact issue.
>
>
> 2D/AWT folks: any other thoughts?
>
> --
> best regards,
> Anthony
>
> On 12/29/2012 17:44, Anton Litvinov wrote:
>> Hello,
>>
>> Please review the following fix for a bug.
>>
>> Bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8005607
>>          https://jbs.oracle.com/bugs/browse/JDK-8005607
>> Webrev: http://cr.openjdk.java.net/~alitvinov/8005607/webrev.00
>>
>> The bug consists in a crash which is caused by a stack overflow for
>> the reason of an infinite recursion in AWT native function
>> J2DXErrHandler() under certain circumstances on 32-bit Linux OS. The
>> fix is based on introduction of the logic, which detects indirect
>> recursive calls to J2DXErrHandler() by means of a simple counter, to
>> J2DXErrHandler() native function. Such a solution requires minimum
>> code changes, does not alter the handler's code significantly and
>> eliminates this bug.
>>
>> Adding 2d-dev at openjdk.java.net e-mail alias to the list of recipients
>> of this letter, because the edited function's name is related to Java
>> 2D area of JDK, despite of the fact that the edited file is located in
>> AWT directory.
>>
>> Thank you,
>> Anton