RFR(XS): 8204652: [aix] fix thread stack allocation, too big stack sizes not recognized

[David Holmes]

Hi Goetz,

On 11/06/2018 1:33 AM, Lindenmaier, Goetz wrote:
> Hi,
> 
> This fixes a crash in thread creation with too big stack sizes
> http://cr.openjdk.java.net/~goetz/wr18/8204652-aixHugeStacksCrash/01/
> Please review.
> 
> For Java and Compiler threads the return code of pthread_attr_setstacksize
>   is overwritten. If pthread_attr_setstacksize does not succeed, a tiny
>   stack size with not enough space for the yellow/red pages is allocated.
>   VM crashes with SIGSEGV right away.

On Linux these are not considered fatal errors (assertion failures in 
debug to catch something unexpected during testing). If setting the 
desired stack fails then it falls back to the default system stack size. 
So only failure of the actual pthread_create is considered an error.

One flaw in that logic, as per the issue you have hit it seems, is that 
if the requested stack size is invalid then we should really then try 
the default stack size for that kind of thread (if the requested size 
was different) and not fallback to the system default. (And we should be 
checking for a guard failure in debug mode.)

On AIX it seems you want any failure to be fatal so in that regard 
adding the missing check of ret seems consistent.

I am a little confused about the actual error though as per your later 
mail. If the stacksize then shifted down to 192K why did the 
pthread_create fail? The log shows a guardsize of 4K so that should be 
fine if it is correct. I understand that the resulting stack may be too 
small for the VM guard pages - but that won't result in a pthread_create 
failure. ???

Cheers,
David

> Best regards,
>    Goetz
>