Problems with 7016797: Hotspot: securely/restrictive load dlls and new API for loading system dlls

[Volker Simonis]

The change "7016797: Hotspot: securely/restrictive load dlls and new
API for loading system dlls"
(http://hg.openjdk.java.net/hsx/hsx21/master/rev/5def270bc147) has
several problems:

1. The first problem is that the initialization of the PSApi library
has been removed from os::init_2():

- // initialize PSAPI or ToolHelp for fatal error handler
- if (win32::is_nt()) _init_psapi();
- else _init_toolhelp();

but not all clients which use its functionality have been updated. The
function '_addr_in_ntdll()' calls
'os::PSApiDll::GetModuleInformation()' without a check if the PSApiDll
module is initialized. This probably hasn't been detected until now
because '_addr_in_ntdll()' is only called if 'UseVectoredExceptions'
is true which is no the case by default. I'd suggest to change
'_addr_in_ntdll()' such that it checks for the availability of the
PSApi dll:

static bool _addr_in_ntdll( address addr )
{
  HMODULE hmod;
  MODULEINFO minfo;

  if (!os::PSApiDll::PSApiAvailable()) {
    // I'm not sure what I should return here. I think PSApi should always be
    // available and thus PSApiAvailable() should always return 'true'.
    // Notice that before HS20 the initialization of the PSApi happened in
    // os::init_2() by calling _init_psapi();
    guarantee(false, "PSApi must be available!");
    return false;
  }

  hmod = GetModuleHandle("NTDLL.DLL");
  if ( hmod == NULL ) return false;
  if ( !os::PSApiDll::GetModuleInformation( GetCurrentProcess(), hmod,
                               &minfo, sizeof(MODULEINFO)) )
    return false;

  if ( (addr >= minfo.lpBaseOfDll) &&
       (addr < (address)((uintptr_t)minfo.lpBaseOfDll +
(uintptr_t)minfo.SizeOfImage)))
    return true;
  else
    return false;
}



2. The second problem is that the detection of whether the HotSpot
runs in a JDK6 or JDK7 context is done at compile time and based on
the compiler version in os_windows.hpp:

// JDK7 requires VS2010
#if _MSC_VER < 1600
#define JDK6_OR_EARLIER 1
#endif

This is a setup which will most probably only work in the current
Oracle build context!

There is already a detection of  "JDK6_OR_EARLIER" in
make/windows/build.make and a better solution would be to just
propagate its value to the compiler predefines as this is done with
other variables like for example JRE_RELEASE_VER

Regards,
Volker