error reporting path fix in java.util.prefs.FileSytemPreferences.lockFile0()

[harry muttart]

Greetings,

I would like to submit a jdk runtime fix on behalf of my employer, Azul 
Systems.

The issue is a long-standing error reporting corner case in the openjdk 
source
base.  The same error exists in jdk6, jdk7, jdk8, and jdk9 sources. The same
change easily applies to all of these source bases.  A recent check of 
the latest
openjdk source base shows this as still an issue.

As I am a complete openjdk process newbie.  Can someone please recommend
the simplest way to make this happen?

Thanks to all for your patience.

Harry Muttart
Azul Systems



The issue...
============

 From the runtime code below, you can see there is a path through
java.util.prefs.FileSytemPreferences.lockFile0() which can result in an
error return with an unset errno field.  On the path where the incoming
argument, shared, is JNI_TRUE, any errno set by an open failure is not
saved (in result[1]).  Then, when the return code from the open is checked,
the code zeroes result[0] but the errno slot remains unset.  Because
return[] is stack allocated, return[1] may contain ANY value, and the
reported errno returned is completely unhelpful.

There is an easy fix.  Capture the errno value after ALL fopen() calls.

An Azul customer tripped this issue.


The source file containing the "fix opportunity" is...
jdk6, jdk7, jdk8)
       jdk/src/solaris/native/java/util/FileSystemPreferences.c
jdk9) jdk/src/java.prefs/unix/native/libprefs/FileSystemPreferences.c


The problem code (from openjdk6 source base) is:

Java_java_util_prefs_FileSystemPreferences_lockFile0(JNIEnv *env,
         jclass thisclass, jstring java_fname, jint permission, jboolean 
shared) {
         const char *fname = JNU_GetStringPlatformChars(env, java_fname, 
JNI_FALSE);
         int fd, rc;
         int result[2];
         jintArray javaResult;
         int old_umask;
         FLOCK fl;

         fl.l_whence = SEEK_SET;
         fl.l_len = 0;
         fl.l_start = 0;
         if (shared == JNI_TRUE) {
             fl.l_type = F_RDLCK;
         } else {
             fl.l_type = F_WRLCK;
         }

         if (shared == JNI_TRUE) {
             fd = open(fname, O_RDONLY, 0);
// NEED FIX HERE:  "result[1] = errno;"
         } else {
             old_umask = umask(0);
             fd = open(fname, O_WRONLY|O_CREAT, permission);
             result[1] = errno;
             umask(old_umask);
         }

         if (fd < 0) {
             result[0] = 0;
         } else {
             rc = fcntl(fd, F_SETLK64, &fl);
             result[1] = errno;
             if (rc < 0) {
                 result[0]= 0;
                 close(fd);
             } else {
               result[0] = fd;
             }
         }
         JNU_ReleaseStringPlatformChars(env, java_fname, fname);
         javaResult = (*env)->NewIntArray(env,2);
         (*env)->SetIntArrayRegion(env, javaResult, 0, 2, result);
         return javaResult;
     }