Review request: White box testing API for HotSpot

[Mikael Gerdin]

Thanks Kris and David for the idea!

I ran into a small issue with this solution:
when compiling the testcase javac can't find wb.jar.

If I put wb.jar in jre/lib/ext javac finds it and the test works but I'm 
not sure if it's Ok to put a jar file in ext on the boot class path 
behind the back of the extension class loader.

Any ideas?

/Mikael

On 2011-12-13 08:04, David Holmes wrote:
> On 13/12/2011 4:48 PM, Krystal Mok wrote:
>> Hi Mikael and David,
>>
>> As an alternative, how about inserting wb.jar into the bootclasspath
>> in Arguments::parse_vm_init_args() when -XX:+EnableWhiteboxAPI is
>> enabled, just like the way alt-rt.jar is inserted when
>> -XX:+AggressiveOpts is given, in the product JDK 6 and 7?
>
> And presumably place wb.jar in jre/lib the way alt-rt.jar is.
>
> That seems like a good suggestion - thanks Kris.
>
> David
>
>> - Kris
>>
>> On Tue, Dec 13, 2011 at 11:16 AM, David Holmes <david.holmes at oracle.com
>> <mailto:david.holmes at oracle.com>> wrote:
>>
>> Hi Mikael,
>>
>> I see why you are dependent on being loaded by the boot loader, but
>> I think using the endorsed mechanism for that is somewhat of a hack
>> - this isn't anything to do with being endorsed it is just a way to
>> get on the bootclasspath without modifying the bootclasspath.
>>
>> I'd like to hear what others think of this.
>>
>> David
>>
>>
>> On 12/12/2011 11:13 PM, Mikael Gerdin wrote:
>>
>> David,
>>
>> On 2011-12-11 12:10, David Holmes wrote:
>>
>> On 9/12/2011 11:42 PM, Mikael Gerdin wrote:
>>
>> On 2011-12-09 05:25, David Holmes wrote:
>>
>> 2. lib/ext jars have the same permissions as
>> bootclasspath, so it
>> should
>> work to place it there.
>>
>>
>> I tried that first (before even discovering the
>> existence of the
>> "endorsed" directory) but when I tried it the Whitebox
>> class didn't get
>> have null as its classloader.
>>
>>
>> Right - lib/ext is loaded by the extensions classloader not the
>> bootstraploader. But this is supposed to have as many
>> privileges as the
>> bootstrap loader:
>>
>>
>> Yes, but does the VM know anything about the ext class loader?
>> Is there any way to check from inside the privilege level of a
>> class? (I
>> suppose I can do some sort of upcall to the JDK to check that but is
>> there a shorter way?)
>>
>> Also, there is one more detail about the boot class loader, see
>> below.
>>
>>
>> "By default, installed optional packages in this standard
>> directory are
>> trusted. That is, they are granted the same privileges as if
>> they were
>> core platform classes (those in rt.jar). This default
>> privilege is
>> specified in the system policy file (in
>> <java-home>/jre/lib/security/__java.policy), but can be
>> overridden for a
>> particular optional package by adding the appropriate policy
>> file entry
>> (see Permissions in the JDK).
>>
>> http://docs.oracle.com/javase/__7/docs/technotes/guides/__extensions/spec.html
>>
>> <http://docs.oracle.com/javase/7/docs/technotes/guides/extensions/spec.html>
>>
>>
>>
>> What is it that requires that wb.jar actually be on the
>> bootclasspath?
>>
>>
>> In order to avoid adding more changes to nativeLookup.cpp i just
>> added
>> the JVM_RegisterWhiteboxMethods to the array of methods explicitly
>> menttioned in this file.
>>
>> The current code checks for the null class loader explicitly here:
>> 156 Handle loader(THREAD,
>> 157
>> instanceKlass::cast(method->__method_holder())->class___loader());
>> 158 if (loader.is_null()) {
>> 159 entry = lookup_special_native(jni___name);
>>
>> I could add another JNINativeMethod array and check for
>> "WhiteBox_registerNatives" on all class loaders.
>> If we go down this way I would have to verify with my security
>> contact
>> that he's okay with dropping the boot class loader requirement.
>>
>> /Mikael
>>
>>
>> Cheers,
>> David
>>
>> Looking at the code in
>> src/share/vm/runtime/__arguments.cpp, the class
>> SysClassPath claims to be responsible for handling the
>> boot class path.
>> I don't see any reference to lib/ext there but I'm not
>> very familiar
>> with how the class loading code actually works.
>>
>> /Mikael
>>
>>
>> Thanks,
>> David
>>
>> On 9/12/2011 1:11 AM, Mikael Gerdin wrote:
>>
>> Hi David,
>> Sorry for the delayed response.
>>
>> On 2011-12-05 07:18, David Holmes wrote:
>>
>> Hi Mikael,
>>
>> On 2/12/2011 8:46 PM, Mikael Gerdin wrote:
>>
>> On 2011-12-02 06:32, David Holmes wrote:
>>
>> I'm a little confused as to where
>> wb.jar ends up when I build
>> hotspot. I
>> see this in a makefile:
>>
>>
>> There are a couple of issues in these
>> make files.
>>
>>
>> 26 WB = wb
>> 27
>> 28 WBSRCDIR =
>> $(GAMMADIR)/src/share/tools/__whitebox/src
>> 29
>> 30 WB_JAR = $(GENERATED)/$(WB).jar
>> 31
>> 32 DEST_WB_JAR =
>> $(JAVA_HOME)/lib/$(WB_JAR)
>>
>> Why JAVA_HOME? That's normally a
>> binary installation of a JDK used
>> for
>> building, not somewhere I expect my
>> build to try and put something.
>> Plus
>> the above will expand to:
>>
>>
>> For example, look in jsig.make. It has a
>> target that copies
>> libjsig to
>> JDK_LIBDIR. JDK_LIBDIR is set up in
>> vm.make to point to
>> JAVA_HOME/jre/lib/[arch]. I was only
>> trying to mimic existing
>> behavior
>> with the "install"-targets in the make
>> files.
>>
>>
>> Our build system is somewhat confusing. The
>> top-level Defs.make will
>> set
>> JAVA_HOME based on ABS_BOOTDIR which in turn
>> is set by BOOTDIR which
>> can
>> be overridden by ALT_BOOTDIR. BOOTDIR,
>> ALT_BOOTDIR and JAVA_HOME are
>> all
>> places where the build expects to find an
>> executable JDK for
>> performing
>> build actions like javac compiles, javah etc.
>>
>> As you point out vm.make then sets
>> JDK_LIBDIR based on JAVA_HOME and
>> that is used by the install_* targets, which
>> are dependencies of the
>> vm.make install target. The vm.make install
>> target is itself invoked
>> from top.make's install target.
>>
>> So when do these install targets actually
>> get used? Other than by a
>> developer on the command-line I don't see
>> these targets actually
>> getting
>> used - and they can't be specified as
>> targets for the top-level
>> Makefile. I suspect these may be relics from
>> a time when you would do
>> something like:
>>
>> JAVA_HOME=/my/local/jdk/to/__test/ make
>> product1 install
>>
>>
>> You are probably correct.
>> Do you want me to modify the make file to more
>> closely mimic the
>> behavior of the other make files and let the
>> legacy "install" target
>> stay or do you want me to just not add an
>> install target?
>>
>>
>> And if I build a full JDK, where
>> does wb.jar end up then?
>>
>>
>> $ find . -name wb.jar
>> ./build/linux-amd64/hotspot/__import/jre/lib/endorsed/wb.jar
>> ./build/linux-amd64/hotspot/__outputdir/linux_amd64___compiler2/generated/wb.jar
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The JDK makefiles that build the
>> j2{sdk,re}-image directories do not
>> pick up the wb.jar file.
>>
>>
>> Ok. So what is the expected build process
>> here such that wb is
>> available
>> for use? Are you expecting the developer to
>> do some kind of "make
>> install"?
>>
>>
>> This is primarily intended for use together with
>> our internal test
>> infrastructure (nightly testing etc.). When
>> running these tests
>> there is
>> already a requirement for a JDK to go together
>> with a VM that we are
>> testing. The same goes for jtreg and the HotSpot
>> tests in the test/
>> subdirectory of the HotSpot repository.
>> So if you want to run tests on your own VM
>> wouldn't you need to use
>> something like "make ALT_EXPORT_PATH=/some/jdk
>> all_fastdebug" do
>> automatically copy your libjvm to a working JDK?
>>
>>
>>
>> I also see in make/Makefile:
>>
>> 370
>> $(EXPORT_JRE_LIB_DIR)/__endorsed/%.jar:
>> $(GEN_DIR)/%.jar
>> 371 $(install-file)
>>
>> Why the endorsed subdirectory? This
>> is nothing to do with an
>> "endorsed
>> standard":
>>
>> http://docs.oracle.com/javase/__6/docs/technotes/guides/__standards/
>> <http://docs.oracle.com/javase/6/docs/technotes/guides/standards/>
>>
>>
>> Because of security requirements and
>> implementation details the
>> Whitebox
>> class must be available on the boot
>> class path.
>> Putting the wb.jar file in the endorsed
>> directory is a quick and
>> easy
>> way to achieve that.
>>
>>
>> Why not just in lib? Or perhaps lib/ext?
>> endorsed just seems to be
>> the
>> least appropriate choice here.
>>
>>
>> Because jars in lib/endorsed are automatically
>> added to the _boot_
>> class
>> path.
>> We could put the jar in jre/lib/ext or jre/lib/
>> or just lib/
>> but then all tests using the API would need
>> -Xbootclasspath as an
>> additional command line flag.
>>
>>
>> And now your usage model has confused me
>> because the above export is
>> done for JPRT builds - right? So you want
>> this to be available in a
>> JPRT
>> bundle, but not in a full JDK build?
>>
>>
>> Nightly testing runs on bundles generated by
>> JPRT, by having the API
>> available in those bundles we can have tests
>> that use this API in our
>> nighly testing.
>> If we want to have this API available when doing
>> a full JDK build we
>> can
>> revisit that particular point in the future
>> since that would need
>> to be
>> synchronized with RE as well.
>>
>>
>> Does this clarify your concerns?
>>
>>
>> Not yet :)
>>
>>
>> I'll keep trying then :)
>>
>> /Mikael
>>
>>
>> Thanks,
>> David
>>
>> /Mikael Gerdin
>>
>>
>> ???
>>
>> Thanks,
>> David
>> -----
>>
>> If the VM crashes after this API
>> has been accessed a note will be
>> written in the hs_err file to
>> signal that the API has been used.
>>
>> Webrev:
>> http://cr.openjdk.java.net/~__stefank/mgerdin/wbapi.0/__webrev/
>> <http://cr.openjdk.java.net/~stefank/mgerdin/wbapi.0/webrev/>
>> (thanks to stefank for hosting
>> my webrev :)
>>
>> CR:
>> I'll file a CR tomorrow.
>>
>> Change comments:
>>
>> make/jprt.properties
>>
>> Add a test target to make sure
>> that the API is available on all
>> supported platforms
>>
>> make/**
>>
>> Makefile changes to build the
>> class sun/hotspot/WhiteBox, put it
>> in a
>> JAR file and copy it to the
>> jre/lib/endorsed directory in the
>> export
>> targets.
>> The BSD makefile changes are not
>> tested since I don't have
>> access to
>> any
>> BSD/OSX machine to test them on.
>>
>> src/share/vm/prims/__nativeLookup.cpp
>>
>> Special-case the method
>> sun/hotspot/WhiteBox/__registerNatives
>> and
>> link it
>> to JVM_RegisterWhiteBoxMethods
>>
>> src/share/vm/prims/whitebox.*
>>
>> The implementation of the white
>> box API. The actual API functions
>> are
>> only examples of what we want to
>> be able to do using the API.
>>
>> src/share/vm/runtime/globals.__hpp
>>
>> Add the command line flag
>>
>> src/share/vm/utilities/__vmError.cpp
>>
>> Print a message in hs_err files
>> when white box API has been used.
>>
>> test/Makefile
>>
>> Add a makefile test target for
>> the white box API test
>>
>> test/sanity/wbapi.java
>>
>> JTreg test to ensure that the
>> API works.
>>
>>
>> Thanks
>> /Mikael Gerdin
>>
>>