RFR: 8134396: Check upper limit of flags setting numbers of stack protection pages.

[Lindenmaier, Goetz]

Hi,

I detected a problem with the values of the flags setting the numbers of stack protection pages.
TestOptionsWithRanges shows that illegal values for red and yellow pages are possible.

In JavaThread::create_stack_guard_pages(),
setting StackRedPages=92233720368547753 and StackYellowPages=1 yields

len = (92233720368547753 + 1) * os::vm_page_size()
    = 0x8000000000000000 * os::vm_page_size()
    = 0

Thus not protecting any pages of the stack.

The check in os::init_2() succeeds, as there the Shadow pages are considered, too, so that the overflowed value is > 0.

On linux, the VM stops with the misleading message
"OpenJDK 64-Bit Server VM warning: Attempt to allocate stack guard pages failed."
On aix, we get assertions and SIGSEGVs.

This fix implements a ConstraintFunc for the three flags involved, checking that no overflow
can happen. This is a very nice new feature!
http://cr.openjdk.java.net/~goetz/webrevs/8134396-StGPg/webrev.01/

Please review this change.  I please need a sponsor.

Alternatively, one could set a smaller upper limit in the range.  This limit would have to be small
enough to fulfill the property for any page size possible on 32 bit systems.

Best regards,
  Goetz.