Review request: White box testing API for HotSpot

[David Holmes]

On 9/12/2011 11:42 PM, Mikael Gerdin wrote:
> On 2011-12-09 05:25, David Holmes wrote:
>> 2. lib/ext jars have the same permissions as bootclasspath, so it should
>> work to place it there.
>
> I tried that first (before even discovering the existence of the
> "endorsed" directory) but when I tried it the Whitebox class didn't get
> have null as its classloader.

Right - lib/ext is loaded by the extensions classloader not the 
bootstraploader. But this is supposed to have as many privileges as the 
bootstrap loader:

"By default, installed optional packages in this standard directory are 
trusted. That is, they are granted the same privileges as if they were 
core platform classes (those in rt.jar). This default privilege is 
specified in the system policy file (in 
<java-home>/jre/lib/security/java.policy), but can be overridden for a 
particular optional package by adding the appropriate policy file entry 
(see Permissions in the JDK).

http://docs.oracle.com/javase/7/docs/technotes/guides/extensions/spec.html

What is it that requires that wb.jar actually be on the bootclasspath?

Cheers,
David

> Looking at the code in src/share/vm/runtime/arguments.cpp, the class
> SysClassPath claims to be responsible for handling the boot class path.
> I don't see any reference to lib/ext there but I'm not very familiar
> with how the class loading code actually works.
>
> /Mikael
>
>>
>> Thanks,
>> David
>>
>> On 9/12/2011 1:11 AM, Mikael Gerdin wrote:
>>> Hi David,
>>> Sorry for the delayed response.
>>>
>>> On 2011-12-05 07:18, David Holmes wrote:
>>>> Hi Mikael,
>>>>
>>>> On 2/12/2011 8:46 PM, Mikael Gerdin wrote:
>>>>> On 2011-12-02 06:32, David Holmes wrote:
>>>>>> I'm a little confused as to where wb.jar ends up when I build
>>>>>> hotspot. I
>>>>>> see this in a makefile:
>>>>>
>>>>> There are a couple of issues in these make files.
>>>>>>
>>>>>> 26 WB = wb
>>>>>> 27
>>>>>> 28 WBSRCDIR = $(GAMMADIR)/src/share/tools/whitebox/src
>>>>>> 29
>>>>>> 30 WB_JAR = $(GENERATED)/$(WB).jar
>>>>>> 31
>>>>>> 32 DEST_WB_JAR = $(JAVA_HOME)/lib/$(WB_JAR)
>>>>>>
>>>>>> Why JAVA_HOME? That's normally a binary installation of a JDK used
>>>>>> for
>>>>>> building, not somewhere I expect my build to try and put something.
>>>>>> Plus
>>>>>> the above will expand to:
>>>>>
>>>>> For example, look in jsig.make. It has a target that copies libjsig to
>>>>> JDK_LIBDIR. JDK_LIBDIR is set up in vm.make to point to
>>>>> JAVA_HOME/jre/lib/[arch]. I was only trying to mimic existing behavior
>>>>> with the "install"-targets in the make files.
>>>>
>>>> Our build system is somewhat confusing. The top-level Defs.make will
>>>> set
>>>> JAVA_HOME based on ABS_BOOTDIR which in turn is set by BOOTDIR which
>>>> can
>>>> be overridden by ALT_BOOTDIR. BOOTDIR, ALT_BOOTDIR and JAVA_HOME are
>>>> all
>>>> places where the build expects to find an executable JDK for performing
>>>> build actions like javac compiles, javah etc.
>>>>
>>>> As you point out vm.make then sets JDK_LIBDIR based on JAVA_HOME and
>>>> that is used by the install_* targets, which are dependencies of the
>>>> vm.make install target. The vm.make install target is itself invoked
>>>> from top.make's install target.
>>>>
>>>> So when do these install targets actually get used? Other than by a
>>>> developer on the command-line I don't see these targets actually
>>>> getting
>>>> used - and they can't be specified as targets for the top-level
>>>> Makefile. I suspect these may be relics from a time when you would do
>>>> something like:
>>>>
>>>> JAVA_HOME=/my/local/jdk/to/test/ make product1 install
>>>
>>> You are probably correct.
>>> Do you want me to modify the make file to more closely mimic the
>>> behavior of the other make files and let the legacy "install" target
>>> stay or do you want me to just not add an install target?
>>>
>>>>
>>>>>> And if I build a full JDK, where does wb.jar end up then?
>>>>>
>>>>> $ find . -name wb.jar
>>>>> ./build/linux-amd64/hotspot/import/jre/lib/endorsed/wb.jar
>>>>> ./build/linux-amd64/hotspot/outputdir/linux_amd64_compiler2/generated/wb.jar
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The JDK makefiles that build the j2{sdk,re}-image directories do not
>>>>> pick up the wb.jar file.
>>>>
>>>> Ok. So what is the expected build process here such that wb is
>>>> available
>>>> for use? Are you expecting the developer to do some kind of "make
>>>> install"?
>>>
>>> This is primarily intended for use together with our internal test
>>> infrastructure (nightly testing etc.). When running these tests there is
>>> already a requirement for a JDK to go together with a VM that we are
>>> testing. The same goes for jtreg and the HotSpot tests in the test/
>>> subdirectory of the HotSpot repository.
>>> So if you want to run tests on your own VM wouldn't you need to use
>>> something like "make ALT_EXPORT_PATH=/some/jdk all_fastdebug" do
>>> automatically copy your libjvm to a working JDK?
>>>
>>>
>>>>
>>>>>> I also see in make/Makefile:
>>>>>>
>>>>>> 370 $(EXPORT_JRE_LIB_DIR)/endorsed/%.jar: $(GEN_DIR)/%.jar
>>>>>> 371 $(install-file)
>>>>>>
>>>>>> Why the endorsed subdirectory? This is nothing to do with an
>>>>>> "endorsed
>>>>>> standard":
>>>>>>
>>>>>> http://docs.oracle.com/javase/6/docs/technotes/guides/standards/
>>>>>
>>>>> Because of security requirements and implementation details the
>>>>> Whitebox
>>>>> class must be available on the boot class path.
>>>>> Putting the wb.jar file in the endorsed directory is a quick and easy
>>>>> way to achieve that.
>>>>
>>>> Why not just in lib? Or perhaps lib/ext? endorsed just seems to be the
>>>> least appropriate choice here.
>>>
>>> Because jars in lib/endorsed are automatically added to the _boot_ class
>>> path.
>>> We could put the jar in jre/lib/ext or jre/lib/ or just lib/
>>> but then all tests using the API would need -Xbootclasspath as an
>>> additional command line flag.
>>>
>>>>
>>>> And now your usage model has confused me because the above export is
>>>> done for JPRT builds - right? So you want this to be available in a
>>>> JPRT
>>>> bundle, but not in a full JDK build?
>>>
>>> Nightly testing runs on bundles generated by JPRT, by having the API
>>> available in those bundles we can have tests that use this API in our
>>> nighly testing.
>>> If we want to have this API available when doing a full JDK build we can
>>> revisit that particular point in the future since that would need to be
>>> synchronized with RE as well.
>>>
>>>>
>>>>> Does this clarify your concerns?
>>>>
>>>> Not yet :)
>>>
>>> I'll keep trying then :)
>>>
>>> /Mikael
>>>
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>>> /Mikael Gerdin
>>>>>
>>>>>>
>>>>>> ???
>>>>>>
>>>>>> Thanks,
>>>>>> David
>>>>>> -----
>>>>>>
>>>>>>> If the VM crashes after this API has been accessed a note will be
>>>>>>> written in the hs_err file to signal that the API has been used.
>>>>>>>
>>>>>>> Webrev:
>>>>>>> http://cr.openjdk.java.net/~stefank/mgerdin/wbapi.0/webrev/
>>>>>>> (thanks to stefank for hosting my webrev :)
>>>>>>>
>>>>>>> CR:
>>>>>>> I'll file a CR tomorrow.
>>>>>>>
>>>>>>> Change comments:
>>>>>>>
>>>>>>> make/jprt.properties
>>>>>>>
>>>>>>> Add a test target to make sure that the API is available on all
>>>>>>> supported platforms
>>>>>>>
>>>>>>> make/**
>>>>>>>
>>>>>>> Makefile changes to build the class sun/hotspot/WhiteBox, put it
>>>>>>> in a
>>>>>>> JAR file and copy it to the jre/lib/endorsed directory in the export
>>>>>>> targets.
>>>>>>> The BSD makefile changes are not tested since I don't have access to
>>>>>>> any
>>>>>>> BSD/OSX machine to test them on.
>>>>>>>
>>>>>>> src/share/vm/prims/nativeLookup.cpp
>>>>>>>
>>>>>>> Special-case the method sun/hotspot/WhiteBox/registerNatives and
>>>>>>> link it
>>>>>>> to JVM_RegisterWhiteBoxMethods
>>>>>>>
>>>>>>> src/share/vm/prims/whitebox.*
>>>>>>>
>>>>>>> The implementation of the white box API. The actual API functions
>>>>>>> are
>>>>>>> only examples of what we want to be able to do using the API.
>>>>>>>
>>>>>>> src/share/vm/runtime/globals.hpp
>>>>>>>
>>>>>>> Add the command line flag
>>>>>>>
>>>>>>> src/share/vm/utilities/vmError.cpp
>>>>>>>
>>>>>>> Print a message in hs_err files when white box API has been used.
>>>>>>>
>>>>>>> test/Makefile
>>>>>>>
>>>>>>> Add a makefile test target for the white box API test
>>>>>>>
>>>>>>> test/sanity/wbapi.java
>>>>>>>
>>>>>>> JTreg test to ensure that the API works.
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> /Mikael Gerdin