strange behavior with stack overflow on windows

[David Holmes]

On 6/06/2013 8:59 PM, Roland Westrelin wrote:
> Hi Andreas,
>
>> IMO, the problem here is a conflict between the hotspot and stack overflow detection and the Windows internal SOF detection:
>>
>> Windows only uses reserved but only partially committed memory for its stacks. In order to detect when to commit more stack, it installs  a one-shot guard page (btw the same type of guard page that is used for the hotspot yellow and red zone) right at the edge of the currently commited stack zone. When a thread accesses this guard page an exception is thrown which Windows catches internally, commits more stack and re-establishes the one-shot guard page at the new edge of the commited zone. When Windows detects such an exception inside the _last 4 pages_ of a stack (I couldn't find any documentation for that on MSDN, I found this value from manually testing on several Windows machines with 4k stack pages) it throws a STACK_OVERFLOW_EXCEPTION.
>>
>> This implies:
>> - If you only have 3 guard pages, a stack overflow will actually occur one page _ahead_ of the yellow zone the first time.
>> - If you have more than 4 guard pages they will have no use because Windows will interpret them as it's normal guard pages used for stack committing and will not throw a STACK_OVERFLOW_EXCEPTION until again the stack has reached the last 4 pages of the stack.
>>
>> Does your problem also occur if you have 4 guard pages? Also, we'll have to test how Windows behaves for pages > 4k...
>
> Thanks for your comments and insight.

Indeed that is a very interesting discovery. Simple testing shows that 
Windows stack handling changes from run to run and within a run. But if 
we force 4 guard pages (red + yellow) we at least get consistent 
behaviour in the running VM.

David
-----

> It doesn't reproduce with StackYellowPages=3.
>
> Roland.
>