JEP 270 concerns

[Andrew Haley]

I've been porting "JEP 270: Reserved Stack Areas for Critical
Sections" to AArch64.  I have no particular concerns about the port,
but I found some serious flaws in testing.

The main problem is that it doesn't work when inlining is enabled.  To
demonstrate this fact, remove "-Inline" and add "-Xss512k" to the
runtime arguments in ReservedStackTest.  (I tried x86_64 and AArch64.)

The first problem is that the stack walking code in
SharedRuntime::look_for_reserved_stack_annotated_method only looks at
stack frames, not at the methods which have been inlined into compiled
methods.  So, if a method marked with a ReservedStackAccess annotation
is inlined, the runtime code will not see the annotation, and the
reserved zone will not be used.

I thought that this was a simple omission, so I changed
look_for_reserved_stack_annotated_method to walk through Scopes
instead of frames, and this indeed detects that a ReservedStackAccess
method has been inlined.  However, there is a deeper flaw: once
control is returned to the method which inlines ReservedStackAccess,
the reserved zone remains disabled, so the next time that a method is
called the protection will not be in place.

I think that the only reasonable way to fix this is to force methods
annotated with ReservedStackAccess not to be inlined.  It would be
possible to fix this in a better way by changing the logic so that a
runtime call to re-enable reserved zone is inserted at the return of
every ReservedStackAccess-annotated method, but this would be more
complex.

There is another problem: if a callee of a ReservedStackAccess method
makes a runtime call, the yellow zone is disabled; when that runtime
call returns, the yellow zone and the reserved zone are re-eanbled, so
the reserved zone protection is re-enabled while that method is
running.  We then return to the ReservedStackAccess method and trigger
an assertion because the reserved zone protection is enabled, which is
unexpected.

Of course I may be very much mistaken about all of this, but I think
not.

Andrew.