RFR(XS): 8204652: [aix] fix thread stack allocation, too big stack sizes not recognized

[Lindenmaier, Goetz]

Hi,

Yes, reporting the pthread_create is a bit misleading. But it is the same 
as reported on linux x86_64:

bin/java -XX:CompilerThreadStackSize=8000000000000
[0.271s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 8000000000000k, guardsize: 0k, detached.
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

As we know linux is bad at implementing the pthread functionality here:
It accepts setting the big value as stack size, and only when you call create
you get the failure.

So it's not true that the VM starts with any other value, it also crashes on 
linux.  And you get the same output as on linux. So I realy don't want to 
start implementing any clever tracing, or even implement other stack
handling (setting defaults) as on linux.  I would just like to keep it as is.
Currently we get a SIGSEGV which is hard to debug on aix.

Best,
  Goetz

> -----Original Message-----
> From: Thomas Stüfe [mailto:thomas.stuefe at gmail.com]
> Sent: Montag, 11. Juni 2018 06:26
> To: Lindenmaier, Goetz <goetz.lindenmaier at sap.com>
> Cc: hotspot-runtime-dev at openjdk.java.net
> Subject: Re: RFR(XS): 8204652: [aix] fix thread stack allocation, too big stack
> sizes not recognized
> 
> Hi Goetz,
> 
> On Sun, Jun 10, 2018 at 9:11 PM, Lindenmaier, Goetz
> <goetz.lindenmaier at sap.com> wrote:
> > Hi Thomas,
> >
> > The stack is started with 192k.  See the output:
> >
> 
> Ah you are right. With 64K pages this would not be enough to establish
> red/yellow zones and still have breathing space left.
> 
> > bash-4.4$ ~/oJ/vms/aix_j-1/bin/java -
> XX:CompilerThreadStackSize=8000000000000
> > [1.482s][warning][os,thread] The compiler thread stack size specified is
> invalid: 8000000000000k
> > [1.483s][warning][os,thread] Failed to start thread - pthread_create failed
> (22=EINVAL) for attributes: stacksize: 192k, guardsize: 4k, detached.
> > Error occurred during initialization of VM
> > java.lang.OutOfMemoryError: unable to create native thread: possibly out
> of memory or process/resource limits reached
> >
> > I think the output is not that bad ...
> 
> See my mail to David: I think the output is misleading, since
> pthread_create() did not even run. So, "pthread_create failed" is
> wrong.
> 
> Best Regards, Thomas
> 
> >
> > Best,
> >   Goetz.
> >
> >> -----Original Message-----
> >> From: Thomas Stüfe <thomas.stuefe at gmail.com>
> >> Sent: Sunday, June 10, 2018 7:33 PM
> >> To: Lindenmaier, Goetz <goetz.lindenmaier at sap.com>
> >> Cc: hotspot-runtime-dev at openjdk.java.net
> >> Subject: Re: RFR(XS): 8204652: [aix] fix thread stack allocation, too big
> stack
> >> sizes not recognized
> >>
> >> Hi Goetz,
> >>
> >> I see what you intent to do, but I think we need to think about error
> >> handling a bit more. The pattern here (which was broken you try to fix
> >> it with this patch) is that we try, sequentially:
> >>
> >> - pthread_attr_setstacksize()
> >>
> >> - if ok-so-far pthread_attr_setguardsize
> >>
> >> - if ok-so-far pthread_ceate
> >>
> >> - logging and error handling
> >>
> >>
> >> Problem with that is that in the end, we log information about a
> >> failing thread creation, printing errno as if pthread_create() failed,
> >> but that errno may have come actually from one of the preceeding OS
> >> calls. So the error message would be misleading.
> >>
> >> I think it would be better to handle errors of each system call explicitly:
> >>
> >> pthread_attr_setstacksize()
> >> if failed
> >>    - log out "pthread_attr_setsize failed" and errno
> >>    - [ also: optionally decide what to do: failing thread creation?
> >> retry with a default stack size? ]
> >>    - return false
> >>
> >> pthread_attr_setguardsize()
> >> if failed
> >>    - log out "pthread_attr_setguardsize failed" and errno
> >>    - decide what to do - ignore? assert? fail thread creation? Should
> >> we even care?
> >>    - return false
> >>
> >> and then call pthread_create with the existing error handling as follow up.
> >>
> >> What do you think?`
> >>
> >> (Also, if pthread_attr_setstacksize() fails, should the thread not be
> >> created with the pthread default stack size, which should be ~2MB? Why
> >> is that stack tiny?)
> >>
> >> Best Regards, Thomas
> >>
> >>
> >> On Sun, Jun 10, 2018 at 5:33 PM, Lindenmaier, Goetz
> >> <goetz.lindenmaier at sap.com> wrote:
> >> > Hi,
> >> >
> >> > This fixes a crash in thread creation with too big stack sizes
> >> > http://cr.openjdk.java.net/~goetz/wr18/8204652-
> aixHugeStacksCrash/01/
> >> > Please review.
> >> >
> >> > For Java and Compiler threads the return code of
> >> pthread_attr_setstacksize
> >> >  is overwritten. If pthread_attr_setstacksize does not succeed, a tiny
> >> >  stack size with not enough space for the yellow/red pages is allocated.
> >> >  VM crashes with SIGSEGV right away.
> >> >
> >> > Best regards,
> >> >   Goetz