[PATCH] Invalid critical JNI function lookup (Windows x86)

[Ioannis Tsakpinis]

This patch fixes the lookup of critical JNI functions on Windows x86.

There are two problems with the argument size calculation in the
lookup_critical_entry function:

1) Critical natives do not have a JNIEnv parameter. Critical natives are
always static, but do not have a jclass parameter. The current code assumes
that both parameters exist and counts them against the total argument size.

2) For each Java array parameter, the critical native gets an additional
length parameter for that array. The current code does not count them.

On the 32-bit VM, the argument size is used to apply stdcall decorations to
the function name. A wrong size is calculated with the current code, so the
name used for the lookup is invalid (unless the function happens to have
exactly two array parameters).

diff -r fec31089c2ef src/share/vm/prims/nativeLookup.cpp
--- a/src/share/vm/prims/nativeLookup.cpp       Thu Oct 06 18:05:53 2016 -0700
+++ b/src/share/vm/prims/nativeLookup.cpp       Sun Oct 09 22:44:54 2016 +0300
@@ -293,10 +293,12 @@
   char* critical_name = critical_jni_name(method);

   // Compute argument size
-  int args_size = 1                             // JNIEnv
-                + (method->is_static() ? 1 : 0) // class for static methods
-                + method->size_of_parameters(); // actual parameters
-
+  int args_size = method->size_of_parameters(); // actual parameters
+  for (SignatureStream ss(signature); !ss.at_return_type(); ss.next()) {
+    if (ss.is_array()) {
+        args_size += T_INT_size; // array length parameter
+    }
+  }

   // 1) Try JNI short style
   entry = lookup_critical_style(method, critical_name, "",
args_size, true);

In steps 3 and 4 the function lookup is done without a prefix/suffix, so a
workaround is available. On msvc JNI functions can be exported without
decorations, but it's not without pain: it requires pragmas or a .DEF file.

Regards,
Ioannis