RFR 12 : 8072130 : java/lang/instrument/BootClassPath/BootClassPathTest.sh fails on Mac OSX

[Brent Christian]

Hi,

Please review this change to how the platform encoding is determined on 
Mac when loading agents.

Webrev: http://cr.openjdk.java.net/~bchristi/8072130/webrev.01.cleanned/

Additional details in the bug report:
https://bugs.openjdk.java.net/browse/JDK-8072130

Some background:
Since JDK 8, if MacOS reports an encoding of US-ASCII, but no encoding 
hints are provided in environment variables (LANG, LC_ALL, LC_CTYPE), 
the JDK uses UTF-8 instead; see JDK-8011194[1].  (This allowed apps 
launched via double-clicked jar to display files correctly for non-ASCII 
languages).

A similar encoding tweak was not made in the code path used for loading 
agents.

This can come into play when an agent jar includes a Boot-Class-Path 
entry containing non-ASCII characters.  iconv() gets setup to convert 
UTF-8 to US-ASCII[2], but will fail when encountering extended 
characters in the entry, so the path in question is not added to the 
bootclasspath[3].  This will likely lead to a ClassNotFoundException.

When java/lang/instrument/BootClassPath/BootClassPathTest.sh is run on a 
  Mac with no values for LANG/LC_ALL/LC_CTYPE, Setup.java sees a default 
  encoding of UTF-8 (per 8011194) and creates Agent.jar with a 
Boot-Class-Path that includes non-ASCII characters.  But the 
Boot-Class-Path entry gets skipped when loading Agent.jar, so the 
AgentSupport class can't be found.

I propose this be fixed by adding the same UTF-8 encoding adjustment on 
MacOS from 8011194 to the agent loading code path.

An additional note:
This bug was challenging to reproduce, in large part because the 8011194 
code checks for an absent env var (getenv() == null), but not for env 
vars with a blank/empty string value (getenv() == ""), so the test 
environment had to be setup in a specific way (e.g. 'unset LANG' vs 
'export LANG=') to reproduce this bug.  So I'd like to update the code 
to also check for empty string.  I also added a couple small 
improvements to the test.

Thanks!

-Brent

1. https://bugs.openjdk.java.net/browse/JDK-8011194
2. 
http://hg.openjdk.java.net/jdk/jdk/file/0517bd2a0eda/src/java.instrument/unix/native/libinstrument/EncodingSupport_md.c#l87
3. 
http://hg.openjdk.java.net/jdk/jdk/file/0517bd2a0eda/src/java.instrument/share/native/libinstrument/InvocationAdapter.c#l873