Review request: White box testing API for HotSpot

[Mikael Gerdin]

I think they will, hopefully they will make this easier.

/Mikael

On 2011-12-14 02:42, John Pampuch wrote:
> Will the modularity changes coming in JDK 8 affect this?
>
> -John
>
> On 12/12/11 11:04 PM, David Holmes wrote:
>> On 13/12/2011 4:48 PM, Krystal Mok wrote:
>>> Hi Mikael and David,
>>>
>>> As an alternative, how about inserting wb.jar into the bootclasspath
>>> in Arguments::parse_vm_init_args() when -XX:+EnableWhiteboxAPI is
>>> enabled, just like the way alt-rt.jar is inserted when
>>> -XX:+AggressiveOpts is given, in the product JDK 6 and 7?
>>
>> And presumably place wb.jar in jre/lib the way alt-rt.jar is.
>>
>> That seems like a good suggestion - thanks Kris.
>>
>> David
>>
>>> - Kris
>>>
>>> On Tue, Dec 13, 2011 at 11:16 AM, David Holmes <david.holmes at oracle.com
>>> <mailto:david.holmes at oracle.com>> wrote:
>>>
>>> Hi Mikael,
>>>
>>> I see why you are dependent on being loaded by the boot loader, but
>>> I think using the endorsed mechanism for that is somewhat of a hack
>>> - this isn't anything to do with being endorsed it is just a way to
>>> get on the bootclasspath without modifying the bootclasspath.
>>>
>>> I'd like to hear what others think of this.
>>>
>>> David
>>>
>>>
>>> On 12/12/2011 11:13 PM, Mikael Gerdin wrote:
>>>
>>> David,
>>>
>>> On 2011-12-11 12:10, David Holmes wrote:
>>>
>>> On 9/12/2011 11:42 PM, Mikael Gerdin wrote:
>>>
>>> On 2011-12-09 05:25, David Holmes wrote:
>>>
>>> 2. lib/ext jars have the same permissions as
>>> bootclasspath, so it
>>> should
>>> work to place it there.
>>>
>>>
>>> I tried that first (before even discovering the
>>> existence of the
>>> "endorsed" directory) but when I tried it the Whitebox
>>> class didn't get
>>> have null as its classloader.
>>>
>>>
>>> Right - lib/ext is loaded by the extensions classloader not the
>>> bootstraploader. But this is supposed to have as many
>>> privileges as the
>>> bootstrap loader:
>>>
>>>
>>> Yes, but does the VM know anything about the ext class loader?
>>> Is there any way to check from inside the privilege level of a
>>> class? (I
>>> suppose I can do some sort of upcall to the JDK to check that but is
>>> there a shorter way?)
>>>
>>> Also, there is one more detail about the boot class loader, see
>>> below.
>>>
>>>
>>> "By default, installed optional packages in this standard
>>> directory are
>>> trusted. That is, they are granted the same privileges as if
>>> they were
>>> core platform classes (those in rt.jar). This default
>>> privilege is
>>> specified in the system policy file (in
>>> <java-home>/jre/lib/security/__java.policy), but can be
>>> overridden for a
>>> particular optional package by adding the appropriate policy
>>> file entry
>>> (see Permissions in the JDK).
>>>
>>> http://docs.oracle.com/javase/__7/docs/technotes/guides/__extensions/spec.html
>>> <http://docs.oracle.com/javase/7/docs/technotes/guides/extensions/spec.html>
>>>
>>>
>>> What is it that requires that wb.jar actually be on the
>>> bootclasspath?
>>>
>>>
>>> In order to avoid adding more changes to nativeLookup.cpp i just
>>> added
>>> the JVM_RegisterWhiteboxMethods to the array of methods explicitly
>>> menttioned in this file.
>>>
>>> The current code checks for the null class loader explicitly here:
>>> 156 Handle loader(THREAD,
>>> 157
>>> instanceKlass::cast(method->__method_holder())->class___loader());
>>> 158 if (loader.is_null()) {
>>> 159 entry = lookup_special_native(jni___name);
>>>
>>> I could add another JNINativeMethod array and check for
>>> "WhiteBox_registerNatives" on all class loaders.
>>> If we go down this way I would have to verify with my security
>>> contact
>>> that he's okay with dropping the boot class loader requirement.
>>>
>>> /Mikael
>>>
>>>
>>> Cheers,
>>> David
>>>
>>> Looking at the code in
>>> src/share/vm/runtime/__arguments.cpp, the class
>>> SysClassPath claims to be responsible for handling the
>>> boot class path.
>>> I don't see any reference to lib/ext there but I'm not
>>> very familiar
>>> with how the class loading code actually works.
>>>
>>> /Mikael
>>>
>>>
>>> Thanks,
>>> David
>>>
>>> On 9/12/2011 1:11 AM, Mikael Gerdin wrote:
>>>
>>> Hi David,
>>> Sorry for the delayed response.
>>>
>>> On 2011-12-05 07:18, David Holmes wrote:
>>>
>>> Hi Mikael,
>>>
>>> On 2/12/2011 8:46 PM, Mikael Gerdin wrote:
>>>
>>> On 2011-12-02 06:32, David Holmes wrote:
>>>
>>> I'm a little confused as to where
>>> wb.jar ends up when I build
>>> hotspot. I
>>> see this in a makefile:
>>>
>>>
>>> There are a couple of issues in these
>>> make files.
>>>
>>>
>>> 26 WB = wb
>>> 27
>>> 28 WBSRCDIR =
>>> $(GAMMADIR)/src/share/tools/__whitebox/src
>>> 29
>>> 30 WB_JAR = $(GENERATED)/$(WB).jar
>>> 31
>>> 32 DEST_WB_JAR =
>>> $(JAVA_HOME)/lib/$(WB_JAR)
>>>
>>> Why JAVA_HOME? That's normally a
>>> binary installation of a JDK used
>>> for
>>> building, not somewhere I expect my
>>> build to try and put something.
>>> Plus
>>> the above will expand to:
>>>
>>>
>>> For example, look in jsig.make. It has a
>>> target that copies
>>> libjsig to
>>> JDK_LIBDIR. JDK_LIBDIR is set up in
>>> vm.make to point to
>>> JAVA_HOME/jre/lib/[arch]. I was only
>>> trying to mimic existing
>>> behavior
>>> with the "install"-targets in the make
>>> files.
>>>
>>>
>>> Our build system is somewhat confusing. The
>>> top-level Defs.make will
>>> set
>>> JAVA_HOME based on ABS_BOOTDIR which in turn
>>> is set by BOOTDIR which
>>> can
>>> be overridden by ALT_BOOTDIR. BOOTDIR,
>>> ALT_BOOTDIR and JAVA_HOME are
>>> all
>>> places where the build expects to find an
>>> executable JDK for
>>> performing
>>> build actions like javac compiles, javah etc.
>>>
>>> As you point out vm.make then sets
>>> JDK_LIBDIR based on JAVA_HOME and
>>> that is used by the install_* targets, which
>>> are dependencies of the
>>> vm.make install target. The vm.make install
>>> target is itself invoked
>>> from top.make's install target.
>>>
>>> So when do these install targets actually
>>> get used? Other than by a
>>> developer on the command-line I don't see
>>> these targets actually
>>> getting
>>> used - and they can't be specified as
>>> targets for the top-level
>>> Makefile. I suspect these may be relics from
>>> a time when you would do
>>> something like:
>>>
>>> JAVA_HOME=/my/local/jdk/to/__test/ make
>>> product1 install
>>>
>>>
>>> You are probably correct.
>>> Do you want me to modify the make file to more
>>> closely mimic the
>>> behavior of the other make files and let the
>>> legacy "install" target
>>> stay or do you want me to just not add an
>>> install target?
>>>
>>>
>>> And if I build a full JDK, where
>>> does wb.jar end up then?
>>>
>>>
>>> $ find . -name wb.jar
>>> ./build/linux-amd64/hotspot/__import/jre/lib/endorsed/wb.jar
>>> ./build/linux-amd64/hotspot/__outputdir/linux_amd64___compiler2/generated/wb.jar
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The JDK makefiles that build the
>>> j2{sdk,re}-image directories do not
>>> pick up the wb.jar file.
>>>
>>>
>>> Ok. So what is the expected build process
>>> here such that wb is
>>> available
>>> for use? Are you expecting the developer to
>>> do some kind of "make
>>> install"?
>>>
>>>
>>> This is primarily intended for use together with
>>> our internal test
>>> infrastructure (nightly testing etc.). When
>>> running these tests
>>> there is
>>> already a requirement for a JDK to go together
>>> with a VM that we are
>>> testing. The same goes for jtreg and the HotSpot
>>> tests in the test/
>>> subdirectory of the HotSpot repository.
>>> So if you want to run tests on your own VM
>>> wouldn't you need to use
>>> something like "make ALT_EXPORT_PATH=/some/jdk
>>> all_fastdebug" do
>>> automatically copy your libjvm to a working JDK?
>>>
>>>
>>>
>>> I also see in make/Makefile:
>>>
>>> 370
>>> $(EXPORT_JRE_LIB_DIR)/__endorsed/%.jar:
>>> $(GEN_DIR)/%.jar
>>> 371 $(install-file)
>>>
>>> Why the endorsed subdirectory? This
>>> is nothing to do with an
>>> "endorsed
>>> standard":
>>>
>>> http://docs.oracle.com/javase/__6/docs/technotes/guides/__standards/
>>> <http://docs.oracle.com/javase/6/docs/technotes/guides/standards/>
>>>
>>>
>>> Because of security requirements and
>>> implementation details the
>>> Whitebox
>>> class must be available on the boot
>>> class path.
>>> Putting the wb.jar file in the endorsed
>>> directory is a quick and
>>> easy
>>> way to achieve that.
>>>
>>>
>>> Why not just in lib? Or perhaps lib/ext?
>>> endorsed just seems to be
>>> the
>>> least appropriate choice here.
>>>
>>>
>>> Because jars in lib/endorsed are automatically
>>> added to the _boot_
>>> class
>>> path.
>>> We could put the jar in jre/lib/ext or jre/lib/
>>> or just lib/
>>> but then all tests using the API would need
>>> -Xbootclasspath as an
>>> additional command line flag.
>>>
>>>
>>> And now your usage model has confused me
>>> because the above export is
>>> done for JPRT builds - right? So you want
>>> this to be available in a
>>> JPRT
>>> bundle, but not in a full JDK build?
>>>
>>>
>>> Nightly testing runs on bundles generated by
>>> JPRT, by having the API
>>> available in those bundles we can have tests
>>> that use this API in our
>>> nighly testing.
>>> If we want to have this API available when doing
>>> a full JDK build we
>>> can
>>> revisit that particular point in the future
>>> since that would need
>>> to be
>>> synchronized with RE as well.
>>>
>>>
>>> Does this clarify your concerns?
>>>
>>>
>>> Not yet :)
>>>
>>>
>>> I'll keep trying then :)
>>>
>>> /Mikael
>>>
>>>
>>> Thanks,
>>> David
>>>
>>> /Mikael Gerdin
>>>
>>>
>>> ???
>>>
>>> Thanks,
>>> David
>>> -----
>>>
>>> If the VM crashes after this API
>>> has been accessed a note will be
>>> written in the hs_err file to
>>> signal that the API has been used.
>>>
>>> Webrev:
>>> http://cr.openjdk.java.net/~__stefank/mgerdin/wbapi.0/__webrev/
>>> <http://cr.openjdk.java.net/~stefank/mgerdin/wbapi.0/webrev/>
>>> (thanks to stefank for hosting
>>> my webrev :)
>>>
>>> CR:
>>> I'll file a CR tomorrow.
>>>
>>> Change comments:
>>>
>>> make/jprt.properties
>>>
>>> Add a test target to make sure
>>> that the API is available on all
>>> supported platforms
>>>
>>> make/**
>>>
>>> Makefile changes to build the
>>> class sun/hotspot/WhiteBox, put it
>>> in a
>>> JAR file and copy it to the
>>> jre/lib/endorsed directory in the
>>> export
>>> targets.
>>> The BSD makefile changes are not
>>> tested since I don't have
>>> access to
>>> any
>>> BSD/OSX machine to test them on.
>>>
>>> src/share/vm/prims/__nativeLookup.cpp
>>>
>>> Special-case the method
>>> sun/hotspot/WhiteBox/__registerNatives
>>> and
>>> link it
>>> to JVM_RegisterWhiteBoxMethods
>>>
>>> src/share/vm/prims/whitebox.*
>>>
>>> The implementation of the white
>>> box API. The actual API functions
>>> are
>>> only examples of what we want to
>>> be able to do using the API.
>>>
>>> src/share/vm/runtime/globals.__hpp
>>>
>>> Add the command line flag
>>>
>>> src/share/vm/utilities/__vmError.cpp
>>>
>>> Print a message in hs_err files
>>> when white box API has been used.
>>>
>>> test/Makefile
>>>
>>> Add a makefile test target for
>>> the white box API test
>>>
>>> test/sanity/wbapi.java
>>>
>>> JTreg test to ensure that the
>>> API works.
>>>
>>>
>>> Thanks
>>> /Mikael Gerdin
>>>
>>>