RFR(S): 7147740: add assertions to check stack alignment on VM entry from generated code (x64)

[Volker Simonis]

Hi,

sorry for reopening this thread after so long time but I've just
realized that this check is very weak. It currently only works because
we build with '-fno-omit-frame-pointer'. The problem is the following:

- GCC 'usually' ensures that stack frames (and thus the SP) are 16-byte aligned.
- however for framless leaf methods GCC doesn't care a lot about the
SP alignment (because nothing has to read/written to/from stack)
- 'os::current_stack_pointer()' is such a frameless leaf method. Its
code looks as follows (if compiled with '-fomit-frame-pointer')

0000000000000000 <os::current_stack_pointer()>:
       0:       48 89 e0                mov    %rsp,%rax
       3:       c3                      retq

- so if current_stack_pointer() will be called from a method with a
16-byte aligned SP it will always return a 8-byte aligned only SP,
because the call to current_stack_pointer() will push the return
address onto the stack and decrement it by 8 bytes.

- if we build with '-fno-omit-frame-pointer', the code for
current_stack_pointer() looks as follows:

0000000000000000 <os::current_stack_pointer()>:
       0:       55                      push   %rbp
       1:       48 89 e5                mov    %rsp,%rbp
       4:       48 89 e0                mov    %rsp,%rax
       7:       5d                      pop    %rbp
       8:       c3                      retq

- the pushing of the FP will decrease the SP by 8 more bytes and we
'magically' have a 16-byte aligned SP again.


The general problem with the approach of calling a helper like
os::verify_stack_alignment() to verify the stack alignment of a method
is that the helper will NOT verify the SP of the current caller (i.e.
what we really want), but the stack alignment of the second calle
(i.e. os::current_stack_pointer() in this case.

The only solution I'm currently seeing is to either make sure that
os::verify_stack_alignment() as well as os::current_stack_pointer()
get inlined into the caller or to pack them both into a macro which
will be expanded into every user. But I'm neither sure if the inlining
solution is feasible (I think we can not 'enforce' inlining) nor
whether the macro solution will always work (packing inline assembly
into macros doesn't seem to be the most elegant and secure approach).

Any comments?

Regards,
Volker

PS: by the way, I discovered this problem while I was trying to
compile the HotSpot with Clang where os::verify_stack_alignment()
always asserted "incorrect stack alignment", but not because the stack
was badly aligned in the caller of os::verify_stack_alignment(). It
was just badly aligned in os::current_stack_pointer(). You can
reproduce the same problem if you compile HotSpot with
'-fomit-frame-pointer'.

On Fri, Feb 24, 2012 at 9:44 AM, Roland Westrelin
<roland.westrelin at oracle.com> wrote:
>> OK. It makes sense. I looked and all related bugs we had are for 64bit VM.
>
> Thanks for the review, Vladimir.
>
> Roland.