[PATCH] Invalid value passed to critical JNI function (System V AMD64)

[Ioannis Tsakpinis]

This patch fixes invalid values passed to critical JNI functions on the
System V AMD64 ABI (tested on Linux and MacOS).

Specifically, this bug occurs when the arguments of a JNI method with 6 or
more parameters are shuffled to match the critical native function order.
The ABI provides 6 registers for passing arguments, so a temporary register
must be used to break the shuffling cycle (handled in ComputeMoveOrder). If
the temp register happens to be used for a 64bit argument, the target
register will receive an invalid value.

The fix is to call set2 when initializing the temp register to rbx. This
marks the VMRegPair as a single physical register, which is important when
copying 64bit values.

diff -r fec31089c2ef src/cpu/x86/vm/sharedRuntime_x86_64.cpp
--- a/src/cpu/x86/vm/sharedRuntime_x86_64.cpp   Thu Oct 06 18:05:53 2016 -0700
+++ b/src/cpu/x86/vm/sharedRuntime_x86_64.cpp   Sun Oct 09 22:44:54 2016 +0300
@@ -2178,7 +2178,7 @@
   // critical natives they are offset down.
   GrowableArray<int> arg_order(2 * total_in_args);
   VMRegPair tmp_vmreg;
-  tmp_vmreg.set1(rbx->as_VMReg());
+  tmp_vmreg.set2(rbx->as_VMReg());

   if (!is_critical_native) {
     for (int i = total_in_args - 1, c_arg = total_c_args - 1; i >= 0;
i--, c_arg--) {

The shuffling implementation on AArch64 is similar and the same fix may be
required there, but I have not tested it.

Regards,
Ioannis