RFR(XS): 8204652: [aix] fix thread stack allocation, too big stack sizes not recognized

[Lindenmaier, Goetz]

Hi Thomas, 

The stack is started with 192k.  See the output:

bash-4.4$ ~/oJ/vms/aix_j-1/bin/java -XX:CompilerThreadStackSize=8000000000000
[1.482s][warning][os,thread] The compiler thread stack size specified is invalid: 8000000000000k
[1.483s][warning][os,thread] Failed to start thread - pthread_create failed (22=EINVAL) for attributes: stacksize: 192k, guardsize: 4k, detached.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

I think the output is not that bad ...

Best,
  Goetz.

> -----Original Message-----
> From: Thomas Stüfe <thomas.stuefe at gmail.com>
> Sent: Sunday, June 10, 2018 7:33 PM
> To: Lindenmaier, Goetz <goetz.lindenmaier at sap.com>
> Cc: hotspot-runtime-dev at openjdk.java.net
> Subject: Re: RFR(XS): 8204652: [aix] fix thread stack allocation, too big stack
> sizes not recognized
> 
> Hi Goetz,
> 
> I see what you intent to do, but I think we need to think about error
> handling a bit more. The pattern here (which was broken you try to fix
> it with this patch) is that we try, sequentially:
> 
> - pthread_attr_setstacksize()
> 
> - if ok-so-far pthread_attr_setguardsize
> 
> - if ok-so-far pthread_ceate
> 
> - logging and error handling
> 
> 
> Problem with that is that in the end, we log information about a
> failing thread creation, printing errno as if pthread_create() failed,
> but that errno may have come actually from one of the preceeding OS
> calls. So the error message would be misleading.
> 
> I think it would be better to handle errors of each system call explicitly:
> 
> pthread_attr_setstacksize()
> if failed
>    - log out "pthread_attr_setsize failed" and errno
>    - [ also: optionally decide what to do: failing thread creation?
> retry with a default stack size? ]
>    - return false
> 
> pthread_attr_setguardsize()
> if failed
>    - log out "pthread_attr_setguardsize failed" and errno
>    - decide what to do - ignore? assert? fail thread creation? Should
> we even care?
>    - return false
> 
> and then call pthread_create with the existing error handling as follow up.
> 
> What do you think?`
> 
> (Also, if pthread_attr_setstacksize() fails, should the thread not be
> created with the pthread default stack size, which should be ~2MB? Why
> is that stack tiny?)
> 
> Best Regards, Thomas
> 
> 
> On Sun, Jun 10, 2018 at 5:33 PM, Lindenmaier, Goetz
> <goetz.lindenmaier at sap.com> wrote:
> > Hi,
> >
> > This fixes a crash in thread creation with too big stack sizes
> > http://cr.openjdk.java.net/~goetz/wr18/8204652-aixHugeStacksCrash/01/
> > Please review.
> >
> > For Java and Compiler threads the return code of
> pthread_attr_setstacksize
> >  is overwritten. If pthread_attr_setstacksize does not succeed, a tiny
> >  stack size with not enough space for the yellow/red pages is allocated.
> >  VM crashes with SIGSEGV right away.
> >
> > Best regards,
> >   Goetz