Review request: White box testing API for HotSpot

Mikael Gerdin mikael.gerdin at oracle.com
Wed Dec 14 04:18:01 PST 2011 

On 2011-12-14 01:48, David Holmes wrote:
> On 13/12/2011 11:48 PM, Mikael Gerdin wrote:
>> Thanks Kris and David for the idea!
>>
>> I ran into a small issue with this solution:
>> when compiling the testcase javac can't find wb.jar.
>
> I think you will need to add wb.jar explicitly to the classpath for
> javac. Or perhaps use the javac from your VM under test with wb enabled?

Yes.
What I meant was that javac won't _automatically_ find the WhiteBox 
class as it does if I put the jar file in ext/ or endorsed/
I changed the jtreg test to use:
  * @run compile -J-XX:+UnlockDiagnosticVMOptions -J-XX:+WhiteBoxAPI 
wbapi.java
  * @run main/othervm -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI wbapi

but it's a bit ugly imo.
I'd rather have it work automatically but since there doesn't seem to be 
a good way to do that I'll have to settle with this.

/Mikael

>
>> If I put wb.jar in jre/lib/ext javac finds it and the test works but I'm
>> not sure if it's Ok to put a jar file in ext on the boot class path
>> behind the back of the extension class loader.
>
> No I would not do that.
>
> David
> -----
>>
>> Any ideas?
>>
>> /Mikael
>>
>> On 2011-12-13 08:04, David Holmes wrote:
>>> On 13/12/2011 4:48 PM, Krystal Mok wrote:
>>>> Hi Mikael and David,
>>>>
>>>> As an alternative, how about inserting wb.jar into the bootclasspath
>>>> in Arguments::parse_vm_init_args() when -XX:+EnableWhiteboxAPI is
>>>> enabled, just like the way alt-rt.jar is inserted when
>>>> -XX:+AggressiveOpts is given, in the product JDK 6 and 7?
>>>
>>> And presumably place wb.jar in jre/lib the way alt-rt.jar is.
>>>
>>> That seems like a good suggestion - thanks Kris.
>>>
>>> David
>>>
>>>> - Kris
>>>>
>>>> On Tue, Dec 13, 2011 at 11:16 AM, David Holmes <david.holmes at oracle.com
>>>> <mailto:david.holmes at oracle.com>> wrote:
>>>>
>>>> Hi Mikael,
>>>>
>>>> I see why you are dependent on being loaded by the boot loader, but
>>>> I think using the endorsed mechanism for that is somewhat of a hack
>>>> - this isn't anything to do with being endorsed it is just a way to
>>>> get on the bootclasspath without modifying the bootclasspath.
>>>>
>>>> I'd like to hear what others think of this.
>>>>
>>>> David
>>>>
>>>>
>>>> On 12/12/2011 11:13 PM, Mikael Gerdin wrote:
>>>>
>>>> David,
>>>>
>>>> On 2011-12-11 12:10, David Holmes wrote:
>>>>
>>>> On 9/12/2011 11:42 PM, Mikael Gerdin wrote:
>>>>
>>>> On 2011-12-09 05:25, David Holmes wrote:
>>>>
>>>> 2. lib/ext jars have the same permissions as
>>>> bootclasspath, so it
>>>> should
>>>> work to place it there.
>>>>
>>>>
>>>> I tried that first (before even discovering the
>>>> existence of the
>>>> "endorsed" directory) but when I tried it the Whitebox
>>>> class didn't get
>>>> have null as its classloader.
>>>>
>>>>
>>>> Right - lib/ext is loaded by the extensions classloader not the
>>>> bootstraploader. But this is supposed to have as many
>>>> privileges as the
>>>> bootstrap loader:
>>>>
>>>>
>>>> Yes, but does the VM know anything about the ext class loader?
>>>> Is there any way to check from inside the privilege level of a
>>>> class? (I
>>>> suppose I can do some sort of upcall to the JDK to check that but is
>>>> there a shorter way?)
>>>>
>>>> Also, there is one more detail about the boot class loader, see
>>>> below.
>>>>
>>>>
>>>> "By default, installed optional packages in this standard
>>>> directory are
>>>> trusted. That is, they are granted the same privileges as if
>>>> they were
>>>> core platform classes (those in rt.jar). This default
>>>> privilege is
>>>> specified in the system policy file (in
>>>> <java-home>/jre/lib/security/__java.policy), but can be
>>>> overridden for a
>>>> particular optional package by adding the appropriate policy
>>>> file entry
>>>> (see Permissions in the JDK).
>>>>
>>>> http://docs.oracle.com/javase/__7/docs/technotes/guides/__extensions/spec.html
>>>>
>>>>
>>>>
>>>> <http://docs.oracle.com/javase/7/docs/technotes/guides/extensions/spec.html>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> What is it that requires that wb.jar actually be on the
>>>> bootclasspath?
>>>>
>>>>
>>>> In order to avoid adding more changes to nativeLookup.cpp i just
>>>> added
>>>> the JVM_RegisterWhiteboxMethods to the array of methods explicitly
>>>> menttioned in this file.
>>>>
>>>> The current code checks for the null class loader explicitly here:
>>>> 156 Handle loader(THREAD,
>>>> 157
>>>> instanceKlass::cast(method->__method_holder())->class___loader());
>>>> 158 if (loader.is_null()) {
>>>> 159 entry = lookup_special_native(jni___name);
>>>>
>>>> I could add another JNINativeMethod array and check for
>>>> "WhiteBox_registerNatives" on all class loaders.
>>>> If we go down this way I would have to verify with my security
>>>> contact
>>>> that he's okay with dropping the boot class loader requirement.
>>>>
>>>> /Mikael
>>>>
>>>>
>>>> Cheers,
>>>> David
>>>>
>>>> Looking at the code in
>>>> src/share/vm/runtime/__arguments.cpp, the class
>>>> SysClassPath claims to be responsible for handling the
>>>> boot class path.
>>>> I don't see any reference to lib/ext there but I'm not
>>>> very familiar
>>>> with how the class loading code actually works.
>>>>
>>>> /Mikael
>>>>
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>> On 9/12/2011 1:11 AM, Mikael Gerdin wrote:
>>>>
>>>> Hi David,
>>>> Sorry for the delayed response.
>>>>
>>>> On 2011-12-05 07:18, David Holmes wrote:
>>>>
>>>> Hi Mikael,
>>>>
>>>> On 2/12/2011 8:46 PM, Mikael Gerdin wrote:
>>>>
>>>> On 2011-12-02 06:32, David Holmes wrote:
>>>>
>>>> I'm a little confused as to where
>>>> wb.jar ends up when I build
>>>> hotspot. I
>>>> see this in a makefile:
>>>>
>>>>
>>>> There are a couple of issues in these
>>>> make files.
>>>>
>>>>
>>>> 26 WB = wb
>>>> 27
>>>> 28 WBSRCDIR =
>>>> $(GAMMADIR)/src/share/tools/__whitebox/src
>>>> 29
>>>> 30 WB_JAR = $(GENERATED)/$(WB).jar
>>>> 31
>>>> 32 DEST_WB_JAR =
>>>> $(JAVA_HOME)/lib/$(WB_JAR)
>>>>
>>>> Why JAVA_HOME? That's normally a
>>>> binary installation of a JDK used
>>>> for
>>>> building, not somewhere I expect my
>>>> build to try and put something.
>>>> Plus
>>>> the above will expand to:
>>>>
>>>>
>>>> For example, look in jsig.make. It has a
>>>> target that copies
>>>> libjsig to
>>>> JDK_LIBDIR. JDK_LIBDIR is set up in
>>>> vm.make to point to
>>>> JAVA_HOME/jre/lib/[arch]. I was only
>>>> trying to mimic existing
>>>> behavior
>>>> with the "install"-targets in the make
>>>> files.
>>>>
>>>>
>>>> Our build system is somewhat confusing. The
>>>> top-level Defs.make will
>>>> set
>>>> JAVA_HOME based on ABS_BOOTDIR which in turn
>>>> is set by BOOTDIR which
>>>> can
>>>> be overridden by ALT_BOOTDIR. BOOTDIR,
>>>> ALT_BOOTDIR and JAVA_HOME are
>>>> all
>>>> places where the build expects to find an
>>>> executable JDK for
>>>> performing
>>>> build actions like javac compiles, javah etc.
>>>>
>>>> As you point out vm.make then sets
>>>> JDK_LIBDIR based on JAVA_HOME and
>>>> that is used by the install_* targets, which
>>>> are dependencies of the
>>>> vm.make install target. The vm.make install
>>>> target is itself invoked
>>>> from top.make's install target.
>>>>
>>>> So when do these install targets actually
>>>> get used? Other than by a
>>>> developer on the command-line I don't see
>>>> these targets actually
>>>> getting
>>>> used - and they can't be specified as
>>>> targets for the top-level
>>>> Makefile. I suspect these may be relics from
>>>> a time when you would do
>>>> something like:
>>>>
>>>> JAVA_HOME=/my/local/jdk/to/__test/ make
>>>> product1 install
>>>>
>>>>
>>>> You are probably correct.
>>>> Do you want me to modify the make file to more
>>>> closely mimic the
>>>> behavior of the other make files and let the
>>>> legacy "install" target
>>>> stay or do you want me to just not add an
>>>> install target?
>>>>
>>>>
>>>> And if I build a full JDK, where
>>>> does wb.jar end up then?
>>>>
>>>>
>>>> $ find . -name wb.jar
>>>> ./build/linux-amd64/hotspot/__import/jre/lib/endorsed/wb.jar
>>>> ./build/linux-amd64/hotspot/__outputdir/linux_amd64___compiler2/generated/wb.jar
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The JDK makefiles that build the
>>>> j2{sdk,re}-image directories do not
>>>> pick up the wb.jar file.
>>>>
>>>>
>>>> Ok. So what is the expected build process
>>>> here such that wb is
>>>> available
>>>> for use? Are you expecting the developer to
>>>> do some kind of "make
>>>> install"?
>>>>
>>>>
>>>> This is primarily intended for use together with
>>>> our internal test
>>>> infrastructure (nightly testing etc.). When
>>>> running these tests
>>>> there is
>>>> already a requirement for a JDK to go together
>>>> with a VM that we are
>>>> testing. The same goes for jtreg and the HotSpot
>>>> tests in the test/
>>>> subdirectory of the HotSpot repository.
>>>> So if you want to run tests on your own VM
>>>> wouldn't you need to use
>>>> something like "make ALT_EXPORT_PATH=/some/jdk
>>>> all_fastdebug" do
>>>> automatically copy your libjvm to a working JDK?
>>>>
>>>>
>>>>
>>>> I also see in make/Makefile:
>>>>
>>>> 370
>>>> $(EXPORT_JRE_LIB_DIR)/__endorsed/%.jar:
>>>> $(GEN_DIR)/%.jar
>>>> 371 $(install-file)
>>>>
>>>> Why the endorsed subdirectory? This
>>>> is nothing to do with an
>>>> "endorsed
>>>> standard":
>>>>
>>>> http://docs.oracle.com/javase/__6/docs/technotes/guides/__standards/
>>>> <http://docs.oracle.com/javase/6/docs/technotes/guides/standards/>
>>>>
>>>>
>>>> Because of security requirements and
>>>> implementation details the
>>>> Whitebox
>>>> class must be available on the boot
>>>> class path.
>>>> Putting the wb.jar file in the endorsed
>>>> directory is a quick and
>>>> easy
>>>> way to achieve that.
>>>>
>>>>
>>>> Why not just in lib? Or perhaps lib/ext?
>>>> endorsed just seems to be
>>>> the
>>>> least appropriate choice here.
>>>>
>>>>
>>>> Because jars in lib/endorsed are automatically
>>>> added to the _boot_
>>>> class
>>>> path.
>>>> We could put the jar in jre/lib/ext or jre/lib/
>>>> or just lib/
>>>> but then all tests using the API would need
>>>> -Xbootclasspath as an
>>>> additional command line flag.
>>>>
>>>>
>>>> And now your usage model has confused me
>>>> because the above export is
>>>> done for JPRT builds - right? So you want
>>>> this to be available in a
>>>> JPRT
>>>> bundle, but not in a full JDK build?
>>>>
>>>>
>>>> Nightly testing runs on bundles generated by
>>>> JPRT, by having the API
>>>> available in those bundles we can have tests
>>>> that use this API in our
>>>> nighly testing.
>>>> If we want to have this API available when doing
>>>> a full JDK build we
>>>> can
>>>> revisit that particular point in the future
>>>> since that would need
>>>> to be
>>>> synchronized with RE as well.
>>>>
>>>>
>>>> Does this clarify your concerns?
>>>>
>>>>
>>>> Not yet :)
>>>>
>>>>
>>>> I'll keep trying then :)
>>>>
>>>> /Mikael
>>>>
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>> /Mikael Gerdin
>>>>
>>>>
>>>> ???
>>>>
>>>> Thanks,
>>>> David
>>>> -----
>>>>
>>>> If the VM crashes after this API
>>>> has been accessed a note will be
>>>> written in the hs_err file to
>>>> signal that the API has been used.
>>>>
>>>> Webrev:
>>>> http://cr.openjdk.java.net/~__stefank/mgerdin/wbapi.0/__webrev/
>>>> <http://cr.openjdk.java.net/~stefank/mgerdin/wbapi.0/webrev/>
>>>> (thanks to stefank for hosting
>>>> my webrev :)
>>>>
>>>> CR:
>>>> I'll file a CR tomorrow.
>>>>
>>>> Change comments:
>>>>
>>>> make/jprt.properties
>>>>
>>>> Add a test target to make sure
>>>> that the API is available on all
>>>> supported platforms
>>>>
>>>> make/**
>>>>
>>>> Makefile changes to build the
>>>> class sun/hotspot/WhiteBox, put it
>>>> in a
>>>> JAR file and copy it to the
>>>> jre/lib/endorsed directory in the
>>>> export
>>>> targets.
>>>> The BSD makefile changes are not
>>>> tested since I don't have
>>>> access to
>>>> any
>>>> BSD/OSX machine to test them on.
>>>>
>>>> src/share/vm/prims/__nativeLookup.cpp
>>>>
>>>> Special-case the method
>>>> sun/hotspot/WhiteBox/__registerNatives
>>>> and
>>>> link it
>>>> to JVM_RegisterWhiteBoxMethods
>>>>
>>>> src/share/vm/prims/whitebox.*
>>>>
>>>> The implementation of the white
>>>> box API. The actual API functions
>>>> are
>>>> only examples of what we want to
>>>> be able to do using the API.
>>>>
>>>> src/share/vm/runtime/globals.__hpp
>>>>
>>>> Add the command line flag
>>>>
>>>> src/share/vm/utilities/__vmError.cpp
>>>>
>>>> Print a message in hs_err files
>>>> when white box API has been used.
>>>>
>>>> test/Makefile
>>>>
>>>> Add a makefile test target for
>>>> the white box API test
>>>>
>>>> test/sanity/wbapi.java
>>>>
>>>> JTreg test to ensure that the
>>>> API works.
>>>>
>>>>
>>>> Thanks
>>>> /Mikael Gerdin
>>>>
>>>>

More information about the hotspot-dev mailing list