Review request: White box testing API for HotSpot

[David Holmes]

Hi Mikael,

I see why you are dependent on being loaded by the boot loader, but I 
think using the endorsed mechanism for that is somewhat of a hack - this 
isn't anything to do with being endorsed it is just a way to get on the 
bootclasspath without modifying the bootclasspath.

I'd like to hear what others think of this.

David

On 12/12/2011 11:13 PM, Mikael Gerdin wrote:
> David,
>
> On 2011-12-11 12:10, David Holmes wrote:
>> On 9/12/2011 11:42 PM, Mikael Gerdin wrote:
>>> On 2011-12-09 05:25, David Holmes wrote:
>>>> 2. lib/ext jars have the same permissions as bootclasspath, so it
>>>> should
>>>> work to place it there.
>>>
>>> I tried that first (before even discovering the existence of the
>>> "endorsed" directory) but when I tried it the Whitebox class didn't get
>>> have null as its classloader.
>>
>> Right - lib/ext is loaded by the extensions classloader not the
>> bootstraploader. But this is supposed to have as many privileges as the
>> bootstrap loader:
>
> Yes, but does the VM know anything about the ext class loader?
> Is there any way to check from inside the privilege level of a class? (I
> suppose I can do some sort of upcall to the JDK to check that but is
> there a shorter way?)
>
> Also, there is one more detail about the boot class loader, see below.
>
>>
>> "By default, installed optional packages in this standard directory are
>> trusted. That is, they are granted the same privileges as if they were
>> core platform classes (those in rt.jar). This default privilege is
>> specified in the system policy file (in
>> <java-home>/jre/lib/security/java.policy), but can be overridden for a
>> particular optional package by adding the appropriate policy file entry
>> (see Permissions in the JDK).
>>
>> http://docs.oracle.com/javase/7/docs/technotes/guides/extensions/spec.html
>>
>>
>> What is it that requires that wb.jar actually be on the bootclasspath?
>
> In order to avoid adding more changes to nativeLookup.cpp i just added
> the JVM_RegisterWhiteboxMethods to the array of methods explicitly
> menttioned in this file.
>
> The current code checks for the null class loader explicitly here:
> 156 Handle loader(THREAD,
> 157 instanceKlass::cast(method->method_holder())->class_loader());
> 158 if (loader.is_null()) {
> 159 entry = lookup_special_native(jni_name);
>
> I could add another JNINativeMethod array and check for
> "WhiteBox_registerNatives" on all class loaders.
> If we go down this way I would have to verify with my security contact
> that he's okay with dropping the boot class loader requirement.
>
> /Mikael
>
>>
>> Cheers,
>> David
>>
>>> Looking at the code in src/share/vm/runtime/arguments.cpp, the class
>>> SysClassPath claims to be responsible for handling the boot class path.
>>> I don't see any reference to lib/ext there but I'm not very familiar
>>> with how the class loading code actually works.
>>>
>>> /Mikael
>>>
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>> On 9/12/2011 1:11 AM, Mikael Gerdin wrote:
>>>>> Hi David,
>>>>> Sorry for the delayed response.
>>>>>
>>>>> On 2011-12-05 07:18, David Holmes wrote:
>>>>>> Hi Mikael,
>>>>>>
>>>>>> On 2/12/2011 8:46 PM, Mikael Gerdin wrote:
>>>>>>> On 2011-12-02 06:32, David Holmes wrote:
>>>>>>>> I'm a little confused as to where wb.jar ends up when I build
>>>>>>>> hotspot. I
>>>>>>>> see this in a makefile:
>>>>>>>
>>>>>>> There are a couple of issues in these make files.
>>>>>>>>
>>>>>>>> 26 WB = wb
>>>>>>>> 27
>>>>>>>> 28 WBSRCDIR = $(GAMMADIR)/src/share/tools/whitebox/src
>>>>>>>> 29
>>>>>>>> 30 WB_JAR = $(GENERATED)/$(WB).jar
>>>>>>>> 31
>>>>>>>> 32 DEST_WB_JAR = $(JAVA_HOME)/lib/$(WB_JAR)
>>>>>>>>
>>>>>>>> Why JAVA_HOME? That's normally a binary installation of a JDK used
>>>>>>>> for
>>>>>>>> building, not somewhere I expect my build to try and put something.
>>>>>>>> Plus
>>>>>>>> the above will expand to:
>>>>>>>
>>>>>>> For example, look in jsig.make. It has a target that copies
>>>>>>> libjsig to
>>>>>>> JDK_LIBDIR. JDK_LIBDIR is set up in vm.make to point to
>>>>>>> JAVA_HOME/jre/lib/[arch]. I was only trying to mimic existing
>>>>>>> behavior
>>>>>>> with the "install"-targets in the make files.
>>>>>>
>>>>>> Our build system is somewhat confusing. The top-level Defs.make will
>>>>>> set
>>>>>> JAVA_HOME based on ABS_BOOTDIR which in turn is set by BOOTDIR which
>>>>>> can
>>>>>> be overridden by ALT_BOOTDIR. BOOTDIR, ALT_BOOTDIR and JAVA_HOME are
>>>>>> all
>>>>>> places where the build expects to find an executable JDK for
>>>>>> performing
>>>>>> build actions like javac compiles, javah etc.
>>>>>>
>>>>>> As you point out vm.make then sets JDK_LIBDIR based on JAVA_HOME and
>>>>>> that is used by the install_* targets, which are dependencies of the
>>>>>> vm.make install target. The vm.make install target is itself invoked
>>>>>> from top.make's install target.
>>>>>>
>>>>>> So when do these install targets actually get used? Other than by a
>>>>>> developer on the command-line I don't see these targets actually
>>>>>> getting
>>>>>> used - and they can't be specified as targets for the top-level
>>>>>> Makefile. I suspect these may be relics from a time when you would do
>>>>>> something like:
>>>>>>
>>>>>> JAVA_HOME=/my/local/jdk/to/test/ make product1 install
>>>>>
>>>>> You are probably correct.
>>>>> Do you want me to modify the make file to more closely mimic the
>>>>> behavior of the other make files and let the legacy "install" target
>>>>> stay or do you want me to just not add an install target?
>>>>>
>>>>>>
>>>>>>>> And if I build a full JDK, where does wb.jar end up then?
>>>>>>>
>>>>>>> $ find . -name wb.jar
>>>>>>> ./build/linux-amd64/hotspot/import/jre/lib/endorsed/wb.jar
>>>>>>> ./build/linux-amd64/hotspot/outputdir/linux_amd64_compiler2/generated/wb.jar
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The JDK makefiles that build the j2{sdk,re}-image directories do not
>>>>>>> pick up the wb.jar file.
>>>>>>
>>>>>> Ok. So what is the expected build process here such that wb is
>>>>>> available
>>>>>> for use? Are you expecting the developer to do some kind of "make
>>>>>> install"?
>>>>>
>>>>> This is primarily intended for use together with our internal test
>>>>> infrastructure (nightly testing etc.). When running these tests
>>>>> there is
>>>>> already a requirement for a JDK to go together with a VM that we are
>>>>> testing. The same goes for jtreg and the HotSpot tests in the test/
>>>>> subdirectory of the HotSpot repository.
>>>>> So if you want to run tests on your own VM wouldn't you need to use
>>>>> something like "make ALT_EXPORT_PATH=/some/jdk all_fastdebug" do
>>>>> automatically copy your libjvm to a working JDK?
>>>>>
>>>>>
>>>>>>
>>>>>>>> I also see in make/Makefile:
>>>>>>>>
>>>>>>>> 370 $(EXPORT_JRE_LIB_DIR)/endorsed/%.jar: $(GEN_DIR)/%.jar
>>>>>>>> 371 $(install-file)
>>>>>>>>
>>>>>>>> Why the endorsed subdirectory? This is nothing to do with an
>>>>>>>> "endorsed
>>>>>>>> standard":
>>>>>>>>
>>>>>>>> http://docs.oracle.com/javase/6/docs/technotes/guides/standards/
>>>>>>>
>>>>>>> Because of security requirements and implementation details the
>>>>>>> Whitebox
>>>>>>> class must be available on the boot class path.
>>>>>>> Putting the wb.jar file in the endorsed directory is a quick and
>>>>>>> easy
>>>>>>> way to achieve that.
>>>>>>
>>>>>> Why not just in lib? Or perhaps lib/ext? endorsed just seems to be
>>>>>> the
>>>>>> least appropriate choice here.
>>>>>
>>>>> Because jars in lib/endorsed are automatically added to the _boot_
>>>>> class
>>>>> path.
>>>>> We could put the jar in jre/lib/ext or jre/lib/ or just lib/
>>>>> but then all tests using the API would need -Xbootclasspath as an
>>>>> additional command line flag.
>>>>>
>>>>>>
>>>>>> And now your usage model has confused me because the above export is
>>>>>> done for JPRT builds - right? So you want this to be available in a
>>>>>> JPRT
>>>>>> bundle, but not in a full JDK build?
>>>>>
>>>>> Nightly testing runs on bundles generated by JPRT, by having the API
>>>>> available in those bundles we can have tests that use this API in our
>>>>> nighly testing.
>>>>> If we want to have this API available when doing a full JDK build we
>>>>> can
>>>>> revisit that particular point in the future since that would need
>>>>> to be
>>>>> synchronized with RE as well.
>>>>>
>>>>>>
>>>>>>> Does this clarify your concerns?
>>>>>>
>>>>>> Not yet :)
>>>>>
>>>>> I'll keep trying then :)
>>>>>
>>>>> /Mikael
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> David
>>>>>>
>>>>>>> /Mikael Gerdin
>>>>>>>
>>>>>>>>
>>>>>>>> ???
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> David
>>>>>>>> -----
>>>>>>>>
>>>>>>>>> If the VM crashes after this API has been accessed a note will be
>>>>>>>>> written in the hs_err file to signal that the API has been used.
>>>>>>>>>
>>>>>>>>> Webrev:
>>>>>>>>> http://cr.openjdk.java.net/~stefank/mgerdin/wbapi.0/webrev/
>>>>>>>>> (thanks to stefank for hosting my webrev :)
>>>>>>>>>
>>>>>>>>> CR:
>>>>>>>>> I'll file a CR tomorrow.
>>>>>>>>>
>>>>>>>>> Change comments:
>>>>>>>>>
>>>>>>>>> make/jprt.properties
>>>>>>>>>
>>>>>>>>> Add a test target to make sure that the API is available on all
>>>>>>>>> supported platforms
>>>>>>>>>
>>>>>>>>> make/**
>>>>>>>>>
>>>>>>>>> Makefile changes to build the class sun/hotspot/WhiteBox, put it
>>>>>>>>> in a
>>>>>>>>> JAR file and copy it to the jre/lib/endorsed directory in the
>>>>>>>>> export
>>>>>>>>> targets.
>>>>>>>>> The BSD makefile changes are not tested since I don't have
>>>>>>>>> access to
>>>>>>>>> any
>>>>>>>>> BSD/OSX machine to test them on.
>>>>>>>>>
>>>>>>>>> src/share/vm/prims/nativeLookup.cpp
>>>>>>>>>
>>>>>>>>> Special-case the method sun/hotspot/WhiteBox/registerNatives and
>>>>>>>>> link it
>>>>>>>>> to JVM_RegisterWhiteBoxMethods
>>>>>>>>>
>>>>>>>>> src/share/vm/prims/whitebox.*
>>>>>>>>>
>>>>>>>>> The implementation of the white box API. The actual API functions
>>>>>>>>> are
>>>>>>>>> only examples of what we want to be able to do using the API.
>>>>>>>>>
>>>>>>>>> src/share/vm/runtime/globals.hpp
>>>>>>>>>
>>>>>>>>> Add the command line flag
>>>>>>>>>
>>>>>>>>> src/share/vm/utilities/vmError.cpp
>>>>>>>>>
>>>>>>>>> Print a message in hs_err files when white box API has been used.
>>>>>>>>>
>>>>>>>>> test/Makefile
>>>>>>>>>
>>>>>>>>> Add a makefile test target for the white box API test
>>>>>>>>>
>>>>>>>>> test/sanity/wbapi.java
>>>>>>>>>
>>>>>>>>> JTreg test to ensure that the API works.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> /Mikael Gerdin