Stack guard pages

[Andrew Haley]

I've been working on a bug in OpenOffice where using Java causes a
later crash.  https://bugzilla.novell.com/show_bug.cgi?id=578802

It's a very strange bug: Java is called, does something, and then the
thread is detached from the Java VM.  Long after, a segfault occurs.

The reason for this crash is the way that stack guard pages work.
When DetachCurrentThread() calls
JavaThread::remove_stack_guard_pages(), the guard pages are not
removed.  So, if at some point later in the process the stack grows
beyond the point where the guard pages were mapped, the Linux kernel
cannot expand the stack any further because the guard pages are in the
way, and a segfault occurs.

I've attached a reproducer for this bug to the end of this message.
It crashes on many of the Linux systems I've tried.

The right way to fix this is for remove_stack_guard_pages() to
munmap() the guard pages.  However, it's essential not to split the
stack region, so if the stack has already grown beyond the guard
pages, we have to unmap() it.  Like so:

bool os::create_stack_guard_pages(char* addr, size_t size) {
  uintptr_t stack_extent, stack_base;
  // get_stack_bounds() returns the memory extent mapped for the stack
  // by the kernel.
  if (get_stack_bounds(&stack_extent, &stack_base)) {
    if (stack_extent < (uintptr_t)addr)
      ::munmap((void*)stack_extent, (uintptr_t)addr - stack_extent);
  }

  return os::commit_memory(addr, size);
}

bool os::remove_stack_guard_pages(char* addr, size_t size) {
  return ::munmap(addr, size) == 0;
}

This fixes the bug.  Unfortunately, there is no os:: interface for
create_stack_guard_pages() and remove_stack_guard_pages(), so this
solution doesn't fit well with the current organization of the VM.  I
can't see any way of making this work only by touching os_linux.?pp .

I could simply patch OpenJDK locally for the Linux distros, but I
think we should have this discussion first.

Andrew.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: invoke.c
Type: text/x-csrc
Size: 1214 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20100223/30bbc56a/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: T.java
Type: text/x-java
Size: 90 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20100223/30bbc56a/attachment-0001.bin