RFR(XS): 8204652: [aix] fix thread stack allocation, too big stack sizes not recognized

[Thomas Stüfe]

On Sun 10. Jun 2018 at 22:39, David Holmes <david.holmes at oracle.com> wrote:

> Hi Goetz,
>
> On 11/06/2018 1:33 AM, Lindenmaier, Goetz wrote:
> > Hi,
> >
> > This fixes a crash in thread creation with too big stack sizes
> > http://cr.openjdk.java.net/~goetz/wr18/8204652-aixHugeStacksCrash/01/
> > Please review.
> >
> > For Java and Compiler threads the return code of
> pthread_attr_setstacksize
> >   is overwritten. If pthread_attr_setstacksize does not succeed, a tiny
> >   stack size with not enough space for the yellow/red pages is allocated.
> >   VM crashes with SIGSEGV right away.
>
> On Linux these are not considered fatal errors (assertion failures in
> debug to catch something unexpected during testing). If setting the
> desired stack fails then it falls back to the default system stack size.
> So only failure of the actual pthread_create is considered an error.
>
> One flaw in that logic, as per the issue you have hit it seems, is that
> if the requested stack size is invalid then we should really then try
> the default stack size for that kind of thread (if the requested size
> was different) and not fallback to the system default. (And we should be
> checking for a guard failure in debug mode.)
>
> On AIX it seems you want any failure to be fatal so in that regard
> adding the missing check of ret seems consistent.
>
> I am a little confused about the actual error though as per your later
> mail. If the stacksize then shifted down to 192K why did the
> pthread_create fail? The log shows a guardsize of 4K so that should be
> fine if it is correct. I understand that the resulting stack may be too
> small for the VM guard pages - but that won't result in a pthread_create
> failure. ???
>
> Cheers,
> David
>

I am quite sure that pthread_create did not run at all. Probably the errno
stems from the failing pthread_attr_setstacksize call, and if the output
was produced by Goetz fixed version, neither pthread_attr_setguardsize nor
pthread_create would have been executed. The error message in this case is
totally misleading. That was the issue I tried to address in my first mail.

..Thomas


> > Best regards,
> >    Goetz
> >
>