Review request: White box testing API for HotSpot

[Mikael Gerdin]

Hi David,

On 2011-12-09 05:25, David Holmes wrote:
> Hi Mikael,
>
> I'll top-post for convenience :)
>
> Two points:
>
> 1. I think it may be simpler to drop the install target - "install"
> seems to be somewhat of a historical relic

Ok

>
> 2. lib/ext jars have the same permissions as bootclasspath, so it should
> work to place it there.

I tried that first (before even discovering the existence of the 
"endorsed" directory) but when I tried it the Whitebox class didn't get 
have null as its classloader.

Looking at the code in src/share/vm/runtime/arguments.cpp, the class 
SysClassPath claims to be responsible for handling the boot class path.
I don't see any reference to lib/ext there but I'm not very familiar 
with how the class loading code actually works.

/Mikael

>
> Thanks,
> David
>
> On 9/12/2011 1:11 AM, Mikael Gerdin wrote:
>> Hi David,
>> Sorry for the delayed response.
>>
>> On 2011-12-05 07:18, David Holmes wrote:
>>> Hi Mikael,
>>>
>>> On 2/12/2011 8:46 PM, Mikael Gerdin wrote:
>>>> On 2011-12-02 06:32, David Holmes wrote:
>>>>> I'm a little confused as to where wb.jar ends up when I build
>>>>> hotspot. I
>>>>> see this in a makefile:
>>>>
>>>> There are a couple of issues in these make files.
>>>>>
>>>>> 26 WB = wb
>>>>> 27
>>>>> 28 WBSRCDIR = $(GAMMADIR)/src/share/tools/whitebox/src
>>>>> 29
>>>>> 30 WB_JAR = $(GENERATED)/$(WB).jar
>>>>> 31
>>>>> 32 DEST_WB_JAR = $(JAVA_HOME)/lib/$(WB_JAR)
>>>>>
>>>>> Why JAVA_HOME? That's normally a binary installation of a JDK used for
>>>>> building, not somewhere I expect my build to try and put something.
>>>>> Plus
>>>>> the above will expand to:
>>>>
>>>> For example, look in jsig.make. It has a target that copies libjsig to
>>>> JDK_LIBDIR. JDK_LIBDIR is set up in vm.make to point to
>>>> JAVA_HOME/jre/lib/[arch]. I was only trying to mimic existing behavior
>>>> with the "install"-targets in the make files.
>>>
>>> Our build system is somewhat confusing. The top-level Defs.make will set
>>> JAVA_HOME based on ABS_BOOTDIR which in turn is set by BOOTDIR which can
>>> be overridden by ALT_BOOTDIR. BOOTDIR, ALT_BOOTDIR and JAVA_HOME are all
>>> places where the build expects to find an executable JDK for performing
>>> build actions like javac compiles, javah etc.
>>>
>>> As you point out vm.make then sets JDK_LIBDIR based on JAVA_HOME and
>>> that is used by the install_* targets, which are dependencies of the
>>> vm.make install target. The vm.make install target is itself invoked
>>> from top.make's install target.
>>>
>>> So when do these install targets actually get used? Other than by a
>>> developer on the command-line I don't see these targets actually getting
>>> used - and they can't be specified as targets for the top-level
>>> Makefile. I suspect these may be relics from a time when you would do
>>> something like:
>>>
>>> JAVA_HOME=/my/local/jdk/to/test/ make product1 install
>>
>> You are probably correct.
>> Do you want me to modify the make file to more closely mimic the
>> behavior of the other make files and let the legacy "install" target
>> stay or do you want me to just not add an install target?
>>
>>>
>>>>> And if I build a full JDK, where does wb.jar end up then?
>>>>
>>>> $ find . -name wb.jar
>>>> ./build/linux-amd64/hotspot/import/jre/lib/endorsed/wb.jar
>>>> ./build/linux-amd64/hotspot/outputdir/linux_amd64_compiler2/generated/wb.jar
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The JDK makefiles that build the j2{sdk,re}-image directories do not
>>>> pick up the wb.jar file.
>>>
>>> Ok. So what is the expected build process here such that wb is available
>>> for use? Are you expecting the developer to do some kind of "make
>>> install"?
>>
>> This is primarily intended for use together with our internal test
>> infrastructure (nightly testing etc.). When running these tests there is
>> already a requirement for a JDK to go together with a VM that we are
>> testing. The same goes for jtreg and the HotSpot tests in the test/
>> subdirectory of the HotSpot repository.
>> So if you want to run tests on your own VM wouldn't you need to use
>> something like "make ALT_EXPORT_PATH=/some/jdk all_fastdebug" do
>> automatically copy your libjvm to a working JDK?
>>
>>
>>>
>>>>> I also see in make/Makefile:
>>>>>
>>>>> 370 $(EXPORT_JRE_LIB_DIR)/endorsed/%.jar: $(GEN_DIR)/%.jar
>>>>> 371 $(install-file)
>>>>>
>>>>> Why the endorsed subdirectory? This is nothing to do with an "endorsed
>>>>> standard":
>>>>>
>>>>> http://docs.oracle.com/javase/6/docs/technotes/guides/standards/
>>>>
>>>> Because of security requirements and implementation details the
>>>> Whitebox
>>>> class must be available on the boot class path.
>>>> Putting the wb.jar file in the endorsed directory is a quick and easy
>>>> way to achieve that.
>>>
>>> Why not just in lib? Or perhaps lib/ext? endorsed just seems to be the
>>> least appropriate choice here.
>>
>> Because jars in lib/endorsed are automatically added to the _boot_ class
>> path.
>> We could put the jar in jre/lib/ext or jre/lib/ or just lib/
>> but then all tests using the API would need -Xbootclasspath as an
>> additional command line flag.
>>
>>>
>>> And now your usage model has confused me because the above export is
>>> done for JPRT builds - right? So you want this to be available in a JPRT
>>> bundle, but not in a full JDK build?
>>
>> Nightly testing runs on bundles generated by JPRT, by having the API
>> available in those bundles we can have tests that use this API in our
>> nighly testing.
>> If we want to have this API available when doing a full JDK build we can
>> revisit that particular point in the future since that would need to be
>> synchronized with RE as well.
>>
>>>
>>>> Does this clarify your concerns?
>>>
>>> Not yet :)
>>
>> I'll keep trying then :)
>>
>> /Mikael
>>
>>>
>>> Thanks,
>>> David
>>>
>>>> /Mikael Gerdin
>>>>
>>>>>
>>>>> ???
>>>>>
>>>>> Thanks,
>>>>> David
>>>>> -----
>>>>>
>>>>>> If the VM crashes after this API has been accessed a note will be
>>>>>> written in the hs_err file to signal that the API has been used.
>>>>>>
>>>>>> Webrev:
>>>>>> http://cr.openjdk.java.net/~stefank/mgerdin/wbapi.0/webrev/
>>>>>> (thanks to stefank for hosting my webrev :)
>>>>>>
>>>>>> CR:
>>>>>> I'll file a CR tomorrow.
>>>>>>
>>>>>> Change comments:
>>>>>>
>>>>>> make/jprt.properties
>>>>>>
>>>>>> Add a test target to make sure that the API is available on all
>>>>>> supported platforms
>>>>>>
>>>>>> make/**
>>>>>>
>>>>>> Makefile changes to build the class sun/hotspot/WhiteBox, put it in a
>>>>>> JAR file and copy it to the jre/lib/endorsed directory in the export
>>>>>> targets.
>>>>>> The BSD makefile changes are not tested since I don't have access to
>>>>>> any
>>>>>> BSD/OSX machine to test them on.
>>>>>>
>>>>>> src/share/vm/prims/nativeLookup.cpp
>>>>>>
>>>>>> Special-case the method sun/hotspot/WhiteBox/registerNatives and
>>>>>> link it
>>>>>> to JVM_RegisterWhiteBoxMethods
>>>>>>
>>>>>> src/share/vm/prims/whitebox.*
>>>>>>
>>>>>> The implementation of the white box API. The actual API functions are
>>>>>> only examples of what we want to be able to do using the API.
>>>>>>
>>>>>> src/share/vm/runtime/globals.hpp
>>>>>>
>>>>>> Add the command line flag
>>>>>>
>>>>>> src/share/vm/utilities/vmError.cpp
>>>>>>
>>>>>> Print a message in hs_err files when white box API has been used.
>>>>>>
>>>>>> test/Makefile
>>>>>>
>>>>>> Add a makefile test target for the white box API test
>>>>>>
>>>>>> test/sanity/wbapi.java
>>>>>>
>>>>>> JTreg test to ensure that the API works.
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> /Mikael Gerdin