Proposal: jtreg tests with native components

Mon Apr 28 08:08:59 UTC 2014

On 25 apr 2014, at 17:31, Jonathan Gibbons <jonathan.gibbons at oracle.com> wrote:

> I'll quibble over the phrase "the same makefile logic".
> 
> I think it is OK to use the same Makefile infrastructure (e.g. the configure.sh mechanism) and the same top level Makefile, but at some level this is going to need to be distinct Makefile logic specific to compiling the code for the tests.

Yes, of course there will be specific makefile logic to compile the code for the tests. What I intended to say was that we can to a significant degree leverage the work that has already been done with the “other” makefiles. For example, there is configure step that sets up all of the environment, there are utility functions that can be used (SetupNativeCompilation in this case) and so on. It just makes sense to compile the tests in the same way.

> I agree with the general concept of pre-building binaries, but it would be good to see the next level of detail:

Agree that there are lots of details that need answers. I have started to develop a proof-of-concept and I can share some of the choices I’ve made below.

> -- where is the source code for the native code of the tests

I choose to put the native source code in the same directory next to the Java test code. That makes it easy to see both the Java and native code at the same time.

I wanted to make it easy to add a new native library and have it compiled, so by default the makefiles picks up any C source files and compiles them into libraries, one library per file. By default, the libraries will be named according to the name of the C file, so a file called libMyTest.c will be compiled into libMyTest.so (or MyTest.dll). This is to make simple things simple.

However, if you need to do something more complicated, you can add a makefile to the directory and that file will be invoked instead. It then has the full power to compile the native code as it sees fit.

> -- is it one library per test, or what

I think we can leave that up to the test. According to the above, there can be any number of C files in a test folder and they will all be compiled. A test can then load (System.loadLibrary()) on or more of these at runtime.

> -- what sort of hierarchy do the libraries end up in.

In my implementation I choose to make this as simple as possible and had all libraries end up in the same output folder. I realize this will create problems if two libraries have the same name. We can avoid this by having a naming convention for test libraries that minimizes conflicts. We can also add makefile logic to detect problems at compile-time. 

Because I use the existing makefile framework, the output folder is specific to the makefile configuration you currently use. I my implementation I have used $JDK_OUTPUTDIR/test/native which translates to something like build/macosx-x86_64-normal-server-release/jdk/test/native/.

Having a single output folder makes it easy to setup the correct java.library.path for a test: they all use the same path.

> I am also concerned for the developer experience.  One of the characteristics of the jtreg design has always been that it is dev-friendly, meaning it is easy and fast for developers to edit a test and rerun it.    For myself, I don't work on native code, or on repos containing native code, so I'd be interested to hear how this will impact developers working on tests, and on those folk that simply want to run the tests.

Yes, this is indeed the largest problem to solve here and I want to keep the feature of simple edit-rerun cycles. On the other hand, I don’t want to complicate jtreg so that it has know anything about how to compile native code or invoke makefiles. I have instead chosen to use the makefile “test” target as the way to run tests and make sure native tests are recompiled. In this case, if you edit a native test you would run “make test” (possibly with some more arguments) and that would compile the test libraries (and the product) and then call jtreg. Because there is no way (that I can think of at least) for jtreg to know where the compiled test libraries are located, the invokation of jtreg will have to include that path as an argument to jtreg.

If you prefer to not use the makefile to run tests, you would have to do two step: first compile native tests (using the makefiles) and then run jtreg manually. You would then have to tell jtreg where the compiled tests are.

This does complicate things, no doubt about that. It’s hard to make it completely transparent and this is the best that I’ve been able to come up with. Other ideas are more than welcome.

Thanks,
/Staffan

> 
> -- Jon
> 
> 
> On 04/25/2014 05:02 AM, Staffan Larsen wrote:
>> There are a couple of jtreg tests today that depend on native components (either JNI libraries or executables). These are handled in one of two ways:
>> 
>> 1) The binaries are pre-compiled and checked into the repository (often inside jar files).
>> 2) The test will try to invoke a compiler (gcc, cl, …) when the test is being run.
>> 
>> Neither of these are very good solutions. #1 makes it hard to run the setup the test for all platforms and requires binaries in the source control system. #2 is hit-and-miss: the correct compiler may or may not be installed on the test machine, and the approach requires platform specific logic to be maintained.
>> 
>> I would like to propose that these native components are instead compiled when the product is built by the same makefile logic as the product. At product build time we know we have access to the (correct) compilers and we have excellent support in the makefiles for building on all platforms.
>> 
>> If we build the native test components together with the product, we also have to take care of distributing the result together with the product when we do testing across a larger number of machines. We will also need a way to tell the jtreg tests where these pre-built binaries are located.
>> 
>> I suggest that at the end of a distributed build run, the pre-built test binaries are packaged in a zip or tar file (just like the product bits) and stored next to the product bundles. When we run distributed tests, we need to pick up the product bundle and the test bundle before the testing is started.
>> 
>> To tell the tests where the native code is, I would like to add a flag to jtreg to point out the path to the binaries. This should cause jtreg to set java.library.path before invoking a test and also set a test.* property which can be used by test to find it’s native components.
>> 
>> This kind of setup would make it easier to add and maintain tests that have a native component. I think this will be especially important as more tests are written using jtreg in the hotspot repository.
>> 
>> Thoughts on this? Is the general approach ok? There are lots of details to be figured out, but at this stage I would like to hear feedback on the idea as such.
>> 
>> Thanks,
>> /Staffan
>> 
> 