Review request: White box testing API for HotSpot

[Joe Darcy]

Hello,

On 12/13/2011 5:48 AM, Mikael Gerdin wrote:
> Thanks Kris and David for the idea!
>
> I ran into a small issue with this solution:
> when compiling the testcase javac can't find wb.jar.
>
> If I put wb.jar in jre/lib/ext javac finds it and the test works but 
> I'm not sure if it's Ok to put a jar file in ext on the boot class 
> path behind the back of the extension class loader.
>
> Any ideas?

I believe this effort would be simplified if the new types were 
unconditionally available in the JDK and not available depending on  
build variations.

The endorsed standards mechanism is meant for other, very specific 
purposes [1].

For a more general discussion about how the full logical classpath is 
constructed in different settings see [2] and [3].

-Joe

[1] 
http://docs.oracle.com/javase/7/docs/technotes/guides/standards/index.html

[2] http://blogs.oracle.com/darcy/entry/pick_a_path_any_path1

[3] http://docs.oracle.com/javase/7/docs/technotes/tools/findingclasses.html


>
> /Mikael
>
> On 2011-12-13 08:04, David Holmes wrote:
>> On 13/12/2011 4:48 PM, Krystal Mok wrote:
>>> Hi Mikael and David,
>>>
>>> As an alternative, how about inserting wb.jar into the bootclasspath
>>> in Arguments::parse_vm_init_args() when -XX:+EnableWhiteboxAPI is
>>> enabled, just like the way alt-rt.jar is inserted when
>>> -XX:+AggressiveOpts is given, in the product JDK 6 and 7?
>>
>> And presumably place wb.jar in jre/lib the way alt-rt.jar is.
>>
>> That seems like a good suggestion - thanks Kris.
>>
>> David
>>
>>> - Kris
>>>
>>> On Tue, Dec 13, 2011 at 11:16 AM, David Holmes <david.holmes at oracle.com
>>> <mailto:david.holmes at oracle.com>> wrote:
>>>
>>> Hi Mikael,
>>>
>>> I see why you are dependent on being loaded by the boot loader, but
>>> I think using the endorsed mechanism for that is somewhat of a hack
>>> - this isn't anything to do with being endorsed it is just a way to
>>> get on the bootclasspath without modifying the bootclasspath.
>>>
>>> I'd like to hear what others think of this.
>>>
>>> David
>>>
>>>
>>> On 12/12/2011 11:13 PM, Mikael Gerdin wrote:
>>>
>>> David,
>>>
>>> On 2011-12-11 12:10, David Holmes wrote:
>>>
>>> On 9/12/2011 11:42 PM, Mikael Gerdin wrote:
>>>
>>> On 2011-12-09 05:25, David Holmes wrote:
>>>
>>> 2. lib/ext jars have the same permissions as
>>> bootclasspath, so it
>>> should
>>> work to place it there.
>>>
>>>
>>> I tried that first (before even discovering the
>>> existence of the
>>> "endorsed" directory) but when I tried it the Whitebox
>>> class didn't get
>>> have null as its classloader.
>>>
>>>
>>> Right - lib/ext is loaded by the extensions classloader not the
>>> bootstraploader. But this is supposed to have as many
>>> privileges as the
>>> bootstrap loader:
>>>
>>>
>>> Yes, but does the VM know anything about the ext class loader?
>>> Is there any way to check from inside the privilege level of a
>>> class? (I
>>> suppose I can do some sort of upcall to the JDK to check that but is
>>> there a shorter way?)
>>>
>>> Also, there is one more detail about the boot class loader, see
>>> below.
>>>
>>>
>>> "By default, installed optional packages in this standard
>>> directory are
>>> trusted. That is, they are granted the same privileges as if
>>> they were
>>> core platform classes (those in rt.jar). This default
>>> privilege is
>>> specified in the system policy file (in
>>> <java-home>/jre/lib/security/__java.policy), but can be
>>> overridden for a
>>> particular optional package by adding the appropriate policy
>>> file entry
>>> (see Permissions in the JDK).
>>>
>>> http://docs.oracle.com/javase/__7/docs/technotes/guides/__extensions/spec.html 
>>>
>>>
>>> <http://docs.oracle.com/javase/7/docs/technotes/guides/extensions/spec.html> 
>>>
>>>
>>>
>>>
>>> What is it that requires that wb.jar actually be on the
>>> bootclasspath?
>>>
>>>
>>> In order to avoid adding more changes to nativeLookup.cpp i just
>>> added
>>> the JVM_RegisterWhiteboxMethods to the array of methods explicitly
>>> menttioned in this file.
>>>
>>> The current code checks for the null class loader explicitly here:
>>> 156 Handle loader(THREAD,
>>> 157
>>> instanceKlass::cast(method->__method_holder())->class___loader());
>>> 158 if (loader.is_null()) {
>>> 159 entry = lookup_special_native(jni___name);
>>>
>>> I could add another JNINativeMethod array and check for
>>> "WhiteBox_registerNatives" on all class loaders.
>>> If we go down this way I would have to verify with my security
>>> contact
>>> that he's okay with dropping the boot class loader requirement.
>>>
>>> /Mikael
>>>
>>>
>>> Cheers,
>>> David
>>>
>>> Looking at the code in
>>> src/share/vm/runtime/__arguments.cpp, the class
>>> SysClassPath claims to be responsible for handling the
>>> boot class path.
>>> I don't see any reference to lib/ext there but I'm not
>>> very familiar
>>> with how the class loading code actually works.
>>>
>>> /Mikael
>>>
>>>
>>> Thanks,
>>> David
>>>
>>> On 9/12/2011 1:11 AM, Mikael Gerdin wrote:
>>>
>>> Hi David,
>>> Sorry for the delayed response.
>>>
>>> On 2011-12-05 07:18, David Holmes wrote:
>>>
>>> Hi Mikael,
>>>
>>> On 2/12/2011 8:46 PM, Mikael Gerdin wrote:
>>>
>>> On 2011-12-02 06:32, David Holmes wrote:
>>>
>>> I'm a little confused as to where
>>> wb.jar ends up when I build
>>> hotspot. I
>>> see this in a makefile:
>>>
>>>
>>> There are a couple of issues in these
>>> make files.
>>>
>>>
>>> 26 WB = wb
>>> 27
>>> 28 WBSRCDIR =
>>> $(GAMMADIR)/src/share/tools/__whitebox/src
>>> 29
>>> 30 WB_JAR = $(GENERATED)/$(WB).jar
>>> 31
>>> 32 DEST_WB_JAR =
>>> $(JAVA_HOME)/lib/$(WB_JAR)
>>>
>>> Why JAVA_HOME? That's normally a
>>> binary installation of a JDK used
>>> for
>>> building, not somewhere I expect my
>>> build to try and put something.
>>> Plus
>>> the above will expand to:
>>>
>>>
>>> For example, look in jsig.make. It has a
>>> target that copies
>>> libjsig to
>>> JDK_LIBDIR. JDK_LIBDIR is set up in
>>> vm.make to point to
>>> JAVA_HOME/jre/lib/[arch]. I was only
>>> trying to mimic existing
>>> behavior
>>> with the "install"-targets in the make
>>> files.
>>>
>>>
>>> Our build system is somewhat confusing. The
>>> top-level Defs.make will
>>> set
>>> JAVA_HOME based on ABS_BOOTDIR which in turn
>>> is set by BOOTDIR which
>>> can
>>> be overridden by ALT_BOOTDIR. BOOTDIR,
>>> ALT_BOOTDIR and JAVA_HOME are
>>> all
>>> places where the build expects to find an
>>> executable JDK for
>>> performing
>>> build actions like javac compiles, javah etc.
>>>
>>> As you point out vm.make then sets
>>> JDK_LIBDIR based on JAVA_HOME and
>>> that is used by the install_* targets, which
>>> are dependencies of the
>>> vm.make install target. The vm.make install
>>> target is itself invoked
>>> from top.make's install target.
>>>
>>> So when do these install targets actually
>>> get used? Other than by a
>>> developer on the command-line I don't see
>>> these targets actually
>>> getting
>>> used - and they can't be specified as
>>> targets for the top-level
>>> Makefile. I suspect these may be relics from
>>> a time when you would do
>>> something like:
>>>
>>> JAVA_HOME=/my/local/jdk/to/__test/ make
>>> product1 install
>>>
>>>
>>> You are probably correct.
>>> Do you want me to modify the make file to more
>>> closely mimic the
>>> behavior of the other make files and let the
>>> legacy "install" target
>>> stay or do you want me to just not add an
>>> install target?
>>>
>>>
>>> And if I build a full JDK, where
>>> does wb.jar end up then?
>>>
>>>
>>> $ find . -name wb.jar
>>> ./build/linux-amd64/hotspot/__import/jre/lib/endorsed/wb.jar
>>> ./build/linux-amd64/hotspot/__outputdir/linux_amd64___compiler2/generated/wb.jar 
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> The JDK makefiles that build the
>>> j2{sdk,re}-image directories do not
>>> pick up the wb.jar file.
>>>
>>>
>>> Ok. So what is the expected build process
>>> here such that wb is
>>> available
>>> for use? Are you expecting the developer to
>>> do some kind of "make
>>> install"?
>>>
>>>
>>> This is primarily intended for use together with
>>> our internal test
>>> infrastructure (nightly testing etc.). When
>>> running these tests
>>> there is
>>> already a requirement for a JDK to go together
>>> with a VM that we are
>>> testing. The same goes for jtreg and the HotSpot
>>> tests in the test/
>>> subdirectory of the HotSpot repository.
>>> So if you want to run tests on your own VM
>>> wouldn't you need to use
>>> something like "make ALT_EXPORT_PATH=/some/jdk
>>> all_fastdebug" do
>>> automatically copy your libjvm to a working JDK?
>>>
>>>
>>>
>>> I also see in make/Makefile:
>>>
>>> 370
>>> $(EXPORT_JRE_LIB_DIR)/__endorsed/%.jar:
>>> $(GEN_DIR)/%.jar
>>> 371 $(install-file)
>>>
>>> Why the endorsed subdirectory? This
>>> is nothing to do with an
>>> "endorsed
>>> standard":
>>>
>>> http://docs.oracle.com/javase/__6/docs/technotes/guides/__standards/
>>> <http://docs.oracle.com/javase/6/docs/technotes/guides/standards/>
>>>
>>>
>>> Because of security requirements and
>>> implementation details the
>>> Whitebox
>>> class must be available on the boot
>>> class path.
>>> Putting the wb.jar file in the endorsed
>>> directory is a quick and
>>> easy
>>> way to achieve that.
>>>
>>>
>>> Why not just in lib? Or perhaps lib/ext?
>>> endorsed just seems to be
>>> the
>>> least appropriate choice here.
>>>
>>>
>>> Because jars in lib/endorsed are automatically
>>> added to the _boot_
>>> class
>>> path.
>>> We could put the jar in jre/lib/ext or jre/lib/
>>> or just lib/
>>> but then all tests using the API would need
>>> -Xbootclasspath as an
>>> additional command line flag.
>>>
>>>
>>> And now your usage model has confused me
>>> because the above export is
>>> done for JPRT builds - right? So you want
>>> this to be available in a
>>> JPRT
>>> bundle, but not in a full JDK build?
>>>
>>>
>>> Nightly testing runs on bundles generated by
>>> JPRT, by having the API
>>> available in those bundles we can have tests
>>> that use this API in our
>>> nighly testing.
>>> If we want to have this API available when doing
>>> a full JDK build we
>>> can
>>> revisit that particular point in the future
>>> since that would need
>>> to be
>>> synchronized with RE as well.
>>>
>>>
>>> Does this clarify your concerns?
>>>
>>>
>>> Not yet :)
>>>
>>>
>>> I'll keep trying then :)
>>>
>>> /Mikael
>>>
>>>
>>> Thanks,
>>> David
>>>
>>> /Mikael Gerdin
>>>
>>>
>>> ???
>>>
>>> Thanks,
>>> David
>>> -----
>>>
>>> If the VM crashes after this API
>>> has been accessed a note will be
>>> written in the hs_err file to
>>> signal that the API has been used.
>>>
>>> Webrev:
>>> http://cr.openjdk.java.net/~__stefank/mgerdin/wbapi.0/__webrev/
>>> <http://cr.openjdk.java.net/~stefank/mgerdin/wbapi.0/webrev/>
>>> (thanks to stefank for hosting
>>> my webrev :)
>>>
>>> CR:
>>> I'll file a CR tomorrow.
>>>
>>> Change comments:
>>>
>>> make/jprt.properties
>>>
>>> Add a test target to make sure
>>> that the API is available on all
>>> supported platforms
>>>
>>> make/**
>>>
>>> Makefile changes to build the
>>> class sun/hotspot/WhiteBox, put it
>>> in a
>>> JAR file and copy it to the
>>> jre/lib/endorsed directory in the
>>> export
>>> targets.
>>> The BSD makefile changes are not
>>> tested since I don't have
>>> access to
>>> any
>>> BSD/OSX machine to test them on.
>>>
>>> src/share/vm/prims/__nativeLookup.cpp
>>>
>>> Special-case the method
>>> sun/hotspot/WhiteBox/__registerNatives
>>> and
>>> link it
>>> to JVM_RegisterWhiteBoxMethods
>>>
>>> src/share/vm/prims/whitebox.*
>>>
>>> The implementation of the white
>>> box API. The actual API functions
>>> are
>>> only examples of what we want to
>>> be able to do using the API.
>>>
>>> src/share/vm/runtime/globals.__hpp
>>>
>>> Add the command line flag
>>>
>>> src/share/vm/utilities/__vmError.cpp
>>>
>>> Print a message in hs_err files
>>> when white box API has been used.
>>>
>>> test/Makefile
>>>
>>> Add a makefile test target for
>>> the white box API test
>>>
>>> test/sanity/wbapi.java
>>>
>>> JTreg test to ensure that the
>>> API works.
>>>
>>>
>>> Thanks
>>> /Mikael Gerdin
>>>
>>>