Review request: White box testing API for HotSpot

Krystal Mok rednaxelafx at gmail.com
Mon Dec 12 22:48:39 PST 2011 

Hi Mikael and David,

As an alternative, how about inserting wb.jar into the bootclasspath
in Arguments::parse_vm_init_args() when -XX:+EnableWhiteboxAPI is enabled,
just like the way alt-rt.jar is inserted when -XX:+AggressiveOpts is given,
in the product JDK 6 and 7?

- Kris

On Tue, Dec 13, 2011 at 11:16 AM, David Holmes <david.holmes at oracle.com>wrote:

> Hi Mikael,
>
> I see why you are dependent on being loaded by the boot loader, but I
> think using the endorsed mechanism for that is somewhat of a hack - this
> isn't anything to do with being endorsed it is just a way to get on the
> bootclasspath without modifying the bootclasspath.
>
> I'd like to hear what others think of this.
>
> David
>
>
> On 12/12/2011 11:13 PM, Mikael Gerdin wrote:
>
>> David,
>>
>> On 2011-12-11 12:10, David Holmes wrote:
>>
>>> On 9/12/2011 11:42 PM, Mikael Gerdin wrote:
>>>
>>>> On 2011-12-09 05:25, David Holmes wrote:
>>>>
>>>>> 2. lib/ext jars have the same permissions as bootclasspath, so it
>>>>> should
>>>>> work to place it there.
>>>>>
>>>>
>>>> I tried that first (before even discovering the existence of the
>>>> "endorsed" directory) but when I tried it the Whitebox class didn't get
>>>> have null as its classloader.
>>>>
>>>
>>> Right - lib/ext is loaded by the extensions classloader not the
>>> bootstraploader. But this is supposed to have as many privileges as the
>>> bootstrap loader:
>>>
>>
>> Yes, but does the VM know anything about the ext class loader?
>> Is there any way to check from inside the privilege level of a class? (I
>> suppose I can do some sort of upcall to the JDK to check that but is
>> there a shorter way?)
>>
>> Also, there is one more detail about the boot class loader, see below.
>>
>>
>>> "By default, installed optional packages in this standard directory are
>>> trusted. That is, they are granted the same privileges as if they were
>>> core platform classes (those in rt.jar). This default privilege is
>>> specified in the system policy file (in
>>> <java-home>/jre/lib/security/**java.policy), but can be overridden for a
>>> particular optional package by adding the appropriate policy file entry
>>> (see Permissions in the JDK).
>>>
>>> http://docs.oracle.com/javase/**7/docs/technotes/guides/**
>>> extensions/spec.html<http://docs.oracle.com/javase/7/docs/technotes/guides/extensions/spec.html>
>>>
>>>
>>> What is it that requires that wb.jar actually be on the bootclasspath?
>>>
>>
>> In order to avoid adding more changes to nativeLookup.cpp i just added
>> the JVM_RegisterWhiteboxMethods to the array of methods explicitly
>> menttioned in this file.
>>
>> The current code checks for the null class loader explicitly here:
>> 156 Handle loader(THREAD,
>> 157 instanceKlass::cast(method->**method_holder())->class_**loader());
>> 158 if (loader.is_null()) {
>> 159 entry = lookup_special_native(jni_**name);
>>
>> I could add another JNINativeMethod array and check for
>> "WhiteBox_registerNatives" on all class loaders.
>> If we go down this way I would have to verify with my security contact
>> that he's okay with dropping the boot class loader requirement.
>>
>> /Mikael
>>
>>
>>> Cheers,
>>> David
>>>
>>>  Looking at the code in src/share/vm/runtime/**arguments.cpp, the class
>>>> SysClassPath claims to be responsible for handling the boot class path.
>>>> I don't see any reference to lib/ext there but I'm not very familiar
>>>> with how the class loading code actually works.
>>>>
>>>> /Mikael
>>>>
>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>> On 9/12/2011 1:11 AM, Mikael Gerdin wrote:
>>>>>
>>>>>> Hi David,
>>>>>> Sorry for the delayed response.
>>>>>>
>>>>>> On 2011-12-05 07:18, David Holmes wrote:
>>>>>>
>>>>>>> Hi Mikael,
>>>>>>>
>>>>>>> On 2/12/2011 8:46 PM, Mikael Gerdin wrote:
>>>>>>>
>>>>>>>> On 2011-12-02 06:32, David Holmes wrote:
>>>>>>>>
>>>>>>>>> I'm a little confused as to where wb.jar ends up when I build
>>>>>>>>> hotspot. I
>>>>>>>>> see this in a makefile:
>>>>>>>>>
>>>>>>>>
>>>>>>>> There are a couple of issues in these make files.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 26 WB = wb
>>>>>>>>> 27
>>>>>>>>> 28 WBSRCDIR = $(GAMMADIR)/src/share/tools/**whitebox/src
>>>>>>>>> 29
>>>>>>>>> 30 WB_JAR = $(GENERATED)/$(WB).jar
>>>>>>>>> 31
>>>>>>>>> 32 DEST_WB_JAR = $(JAVA_HOME)/lib/$(WB_JAR)
>>>>>>>>>
>>>>>>>>> Why JAVA_HOME? That's normally a binary installation of a JDK used
>>>>>>>>> for
>>>>>>>>> building, not somewhere I expect my build to try and put something.
>>>>>>>>> Plus
>>>>>>>>> the above will expand to:
>>>>>>>>>
>>>>>>>>
>>>>>>>> For example, look in jsig.make. It has a target that copies
>>>>>>>> libjsig to
>>>>>>>> JDK_LIBDIR. JDK_LIBDIR is set up in vm.make to point to
>>>>>>>> JAVA_HOME/jre/lib/[arch]. I was only trying to mimic existing
>>>>>>>> behavior
>>>>>>>> with the "install"-targets in the make files.
>>>>>>>>
>>>>>>>
>>>>>>> Our build system is somewhat confusing. The top-level Defs.make will
>>>>>>> set
>>>>>>> JAVA_HOME based on ABS_BOOTDIR which in turn is set by BOOTDIR which
>>>>>>> can
>>>>>>> be overridden by ALT_BOOTDIR. BOOTDIR, ALT_BOOTDIR and JAVA_HOME are
>>>>>>> all
>>>>>>> places where the build expects to find an executable JDK for
>>>>>>> performing
>>>>>>> build actions like javac compiles, javah etc.
>>>>>>>
>>>>>>> As you point out vm.make then sets JDK_LIBDIR based on JAVA_HOME and
>>>>>>> that is used by the install_* targets, which are dependencies of the
>>>>>>> vm.make install target. The vm.make install target is itself invoked
>>>>>>> from top.make's install target.
>>>>>>>
>>>>>>> So when do these install targets actually get used? Other than by a
>>>>>>> developer on the command-line I don't see these targets actually
>>>>>>> getting
>>>>>>> used - and they can't be specified as targets for the top-level
>>>>>>> Makefile. I suspect these may be relics from a time when you would do
>>>>>>> something like:
>>>>>>>
>>>>>>> JAVA_HOME=/my/local/jdk/to/**test/ make product1 install
>>>>>>>
>>>>>>
>>>>>> You are probably correct.
>>>>>> Do you want me to modify the make file to more closely mimic the
>>>>>> behavior of the other make files and let the legacy "install" target
>>>>>> stay or do you want me to just not add an install target?
>>>>>>
>>>>>>
>>>>>>>  And if I build a full JDK, where does wb.jar end up then?
>>>>>>>>>
>>>>>>>>
>>>>>>>> $ find . -name wb.jar
>>>>>>>> ./build/linux-amd64/hotspot/**import/jre/lib/endorsed/wb.jar
>>>>>>>> ./build/linux-amd64/hotspot/**outputdir/linux_amd64_**
>>>>>>>> compiler2/generated/wb.jar
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The JDK makefiles that build the j2{sdk,re}-image directories do not
>>>>>>>> pick up the wb.jar file.
>>>>>>>>
>>>>>>>
>>>>>>> Ok. So what is the expected build process here such that wb is
>>>>>>> available
>>>>>>> for use? Are you expecting the developer to do some kind of "make
>>>>>>> install"?
>>>>>>>
>>>>>>
>>>>>> This is primarily intended for use together with our internal test
>>>>>> infrastructure (nightly testing etc.). When running these tests
>>>>>> there is
>>>>>> already a requirement for a JDK to go together with a VM that we are
>>>>>> testing. The same goes for jtreg and the HotSpot tests in the test/
>>>>>> subdirectory of the HotSpot repository.
>>>>>> So if you want to run tests on your own VM wouldn't you need to use
>>>>>> something like "make ALT_EXPORT_PATH=/some/jdk all_fastdebug" do
>>>>>> automatically copy your libjvm to a working JDK?
>>>>>>
>>>>>>
>>>>>>
>>>>>>>  I also see in make/Makefile:
>>>>>>>>>
>>>>>>>>> 370 $(EXPORT_JRE_LIB_DIR)/**endorsed/%.jar: $(GEN_DIR)/%.jar
>>>>>>>>> 371 $(install-file)
>>>>>>>>>
>>>>>>>>> Why the endorsed subdirectory? This is nothing to do with an
>>>>>>>>> "endorsed
>>>>>>>>> standard":
>>>>>>>>>
>>>>>>>>> http://docs.oracle.com/javase/**6/docs/technotes/guides/**
>>>>>>>>> standards/<http://docs.oracle.com/javase/6/docs/technotes/guides/standards/>
>>>>>>>>>
>>>>>>>>
>>>>>>>> Because of security requirements and implementation details the
>>>>>>>> Whitebox
>>>>>>>> class must be available on the boot class path.
>>>>>>>> Putting the wb.jar file in the endorsed directory is a quick and
>>>>>>>> easy
>>>>>>>> way to achieve that.
>>>>>>>>
>>>>>>>
>>>>>>> Why not just in lib? Or perhaps lib/ext? endorsed just seems to be
>>>>>>> the
>>>>>>> least appropriate choice here.
>>>>>>>
>>>>>>
>>>>>> Because jars in lib/endorsed are automatically added to the _boot_
>>>>>> class
>>>>>> path.
>>>>>> We could put the jar in jre/lib/ext or jre/lib/ or just lib/
>>>>>> but then all tests using the API would need -Xbootclasspath as an
>>>>>> additional command line flag.
>>>>>>
>>>>>>
>>>>>>> And now your usage model has confused me because the above export is
>>>>>>> done for JPRT builds - right? So you want this to be available in a
>>>>>>> JPRT
>>>>>>> bundle, but not in a full JDK build?
>>>>>>>
>>>>>>
>>>>>> Nightly testing runs on bundles generated by JPRT, by having the API
>>>>>> available in those bundles we can have tests that use this API in our
>>>>>> nighly testing.
>>>>>> If we want to have this API available when doing a full JDK build we
>>>>>> can
>>>>>> revisit that particular point in the future since that would need
>>>>>> to be
>>>>>> synchronized with RE as well.
>>>>>>
>>>>>>
>>>>>>>  Does this clarify your concerns?
>>>>>>>>
>>>>>>>
>>>>>>> Not yet :)
>>>>>>>
>>>>>>
>>>>>> I'll keep trying then :)
>>>>>>
>>>>>> /Mikael
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> David
>>>>>>>
>>>>>>>  /Mikael Gerdin
>>>>>>>>
>>>>>>>>
>>>>>>>>> ???
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> David
>>>>>>>>> -----
>>>>>>>>>
>>>>>>>>>  If the VM crashes after this API has been accessed a note will be
>>>>>>>>>> written in the hs_err file to signal that the API has been used.
>>>>>>>>>>
>>>>>>>>>> Webrev:
>>>>>>>>>> http://cr.openjdk.java.net/~**stefank/mgerdin/wbapi.0/**webrev/<http://cr.openjdk.java.net/~stefank/mgerdin/wbapi.0/webrev/>
>>>>>>>>>> (thanks to stefank for hosting my webrev :)
>>>>>>>>>>
>>>>>>>>>> CR:
>>>>>>>>>> I'll file a CR tomorrow.
>>>>>>>>>>
>>>>>>>>>> Change comments:
>>>>>>>>>>
>>>>>>>>>> make/jprt.properties
>>>>>>>>>>
>>>>>>>>>> Add a test target to make sure that the API is available on all
>>>>>>>>>> supported platforms
>>>>>>>>>>
>>>>>>>>>> make/**
>>>>>>>>>>
>>>>>>>>>> Makefile changes to build the class sun/hotspot/WhiteBox, put it
>>>>>>>>>> in a
>>>>>>>>>> JAR file and copy it to the jre/lib/endorsed directory in the
>>>>>>>>>> export
>>>>>>>>>> targets.
>>>>>>>>>> The BSD makefile changes are not tested since I don't have
>>>>>>>>>> access to
>>>>>>>>>> any
>>>>>>>>>> BSD/OSX machine to test them on.
>>>>>>>>>>
>>>>>>>>>> src/share/vm/prims/**nativeLookup.cpp
>>>>>>>>>>
>>>>>>>>>> Special-case the method sun/hotspot/WhiteBox/**registerNatives
>>>>>>>>>> and
>>>>>>>>>> link it
>>>>>>>>>> to JVM_RegisterWhiteBoxMethods
>>>>>>>>>>
>>>>>>>>>> src/share/vm/prims/whitebox.*
>>>>>>>>>>
>>>>>>>>>> The implementation of the white box API. The actual API functions
>>>>>>>>>> are
>>>>>>>>>> only examples of what we want to be able to do using the API.
>>>>>>>>>>
>>>>>>>>>> src/share/vm/runtime/globals.**hpp
>>>>>>>>>>
>>>>>>>>>> Add the command line flag
>>>>>>>>>>
>>>>>>>>>> src/share/vm/utilities/**vmError.cpp
>>>>>>>>>>
>>>>>>>>>> Print a message in hs_err files when white box API has been used.
>>>>>>>>>>
>>>>>>>>>> test/Makefile
>>>>>>>>>>
>>>>>>>>>> Add a makefile test target for the white box API test
>>>>>>>>>>
>>>>>>>>>> test/sanity/wbapi.java
>>>>>>>>>>
>>>>>>>>>> JTreg test to ensure that the API works.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> /Mikael Gerdin
>>>>>>>>>>
>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-dev/attachments/20111213/fe5b9427/attachment-0001.html

More information about the hotspot-dev mailing list