Test Libraries : Status and Further Steps

Tue Oct 2 23:26:32 UTC 2018

Dear all,

I am pleased to announce that we have finished work on 'Merge jdk and hotspot test libraries' and achieved the big milestone in our work on improving test libraries. Since this project started a long time, I think it's worth it to reiterate why we are doing it, what has been done, and what is planned to be done. So by this email, I hope to bring us all on the same page as well as to initiate discussion on possible further improvements.

1. Background and Past Work

The project on merging test libraries started more than three years ago. The test libraries were created long before a consolidated repository, so HotSpot test suite and JDK test suite were not only in different directories but also in separate repositories, and if it was not enough, their developments were happening in different forests. All these were making it really hard to share test libraries and have them synchronized. However, people who work on both hotspot and JDK tests craved to have a consistent test library which covers most common use-cases, this is how several internal forks of test library emerged[1-2]. Obviously, it did not end well: test libraries started to grow differently, and the efforts to keep them in sync became more and more cumbersome. At some point in time, we accepted the forks being different and stopped any attempts to synchronize them and decided to introduce a common library which can be used by all test suites.

The common library had been placed in '/test/lib' directory, which was in the top-level repository, and was based on HotSpot's fork (by the right of the "firstborn"). Christian made a huge and very valuable effort[3-4] on switching all HotSpot tests to use this library.  Although that significantly increased sanity of HotSpot developers and simplified their life, it did not solve the original problem, as we still had multiple copies of a test library. By the end of JDK 9 time frame, we had three forks of test library: two deprecated in JAXP and JDK repositories and one in the top repository. As the top-level library was based on HotSpot' version, switching other test suites required more work and testing.

8075327[5] was created to switch JAXP and JDK test suites to use the common test library. It appeared that only a few JAXP tests were using JAXP version of the test library, so it was relatively easy and straightforward. JDK test library, on the contrary, was heavily used by the test suite. And as JDK test library had been forked back in 2013, it altered a lot from HotSpot's version: some functionality was missed in one version, some in another, some were implemented very differently, etc. Therefore the patch for JDK was not as easy as a simple change of @library/@build tags and import statements and became a long and tedious process of handling one test library class after another. 8211171[6-7] removed JarUtils and "handled" the last class existing in multiple versions of the common test libraries.

2. Future work

Further improvements of the test libraries include handling the rest of JDK and JAXP test library classes; merging logical duplications within the test libraries; modularize the test libraries. Besides these changes, there are other possible enhancements and bug fixes in the test libraries, but they are out of the scope of this project. The following sections explain each of these three improvements in greater details.

2.1. Handling the rest of JDK and JAXP test libraries

Currently, neither JDK nor JAXP test library contains classes from the common test library, but these test libraries still have some other classes which, in some cases, should be rearranged to improve the overall experience with the test libraries.

One might suggest moving all these classes into the top-level library, but such an action would come at a high price. Placing classes into the top-level test library implies certain difficulties: you must assume that they are used by tests from all test suites, so all changes must be reviewed and tested accordingly, and the changes which break behavioral compatibility must be done with great care. Generally speaking, the further library classes away from tests, the greater risk to break other tests.

Leaving classes as-is is another extreme and can also cause some complications. One such complication is a possibility to get a fork of library classes, which will move us back into 2013. Another is a creation of "unintended" test libraries. "Unintended" test library can be created because it is possible to use classes from any other tests (including from another test suite). Although, it is technically possible, expecting someone to test or fix someone else's tests when they change their internal classes would be unreasonable. Thus using classes which are not part of any test libraries and not part of the tests must be discouraged by guidelines and indicated during the review process.

Thereby, the rest of the test libraries classes can be split into three categories and placed accordingly:
 1) classes which are intended to be used by all test suites are to be moved to the top-level test library;
 2) classes which are used by a certain test group and would have very low, if any, usefulness for other test suites. Such classes should be kept in a test library specific for a test suite.
 3) classes which are very specific to a few particular tests should be moved closer to the tests which depend on them.

I have looked through the remaining test library classes, made an initial assessment of their category, and wrote it done as comments in 8211290[8]. Since I am not an original author of these classes nor their user, I might be completely wrong in my evaluation, so I would really appreciate Core-Libs and JAXP team's opinion on them, and would also love them to check whenever I missed any other test library classes.

2.2. Merging duplications

Due to the fact that the top-level test library has been created as a merge of commonly used test library classes, it has several classes which do a similar or the same thing but slightly differently. For example, there are two ways to get a reproducible random generator -- 'jdk.test.lib.Utils.RANDOM_GENERATOR' and 'jdk.test.lib.RandomFactory'. Such duplications have obvious negative impacts on test libraries. All cases of such duplication should be carefully reviewed and "merged" into one entity if possible. Here "merge" doesn't necessarily mean to get a class which provides both a set of APIs and "merge" can be achieved by removing one of the classes accompanied by conversions of the existing tests.   

Poor documentation and visibility of available test library classes seem to be the key causing factors of such duplications. Therefore this effort also includes documenting public classes of the test libraries and making this documentation easily available for Open JDK developers.

8211289[9] has been created to track internal merges and documentation effort. I made an initial identification of duplications and posted a proposed solution for each found case in the RFE.

2.3. Modularize

Analyzing test library classes, one can notice that they are different in their dependencies and/or objectives. Thus it is logical to split them into separate entities and Jigsaw modules are a perfect fit for that. Using Jigsaw modules will allow us to express dependencies more easily and consistently. Although most of the test library depends only on the exported APIs of 'java.base' module, there are classes which have dependencies on other modules as well as classes which use internal packages, so it is logical to use this information to get the first grouping, otherwise, we will needlessly restrict usage of the test library. For example, 'jdk.test.lib.compiler.*' classes are the only classes in the top-level test library which depend on 'java.compiler' module, so it makes sense to place them into a dedicated module and name this module 'test.compiler' to express this dependency. The top-level classes which depend only on 'java.base' module and do not require special handling will be placed into one module which can be named 'test.base'.

Since jtreg supports building Jigsaw modules natively, modularization will also guarantee that all classes are built by a simple and straightforward @build tag. Continuing with the example, tests which want to use 'test.base' module will have to have just two directives '@library /test/lib' and '@build test.base/*' to always get all the needed classes properly built. 

It is important to highlight that although modularization of the test library will make it possible to use a test library as a set of jigsaw modules, it still will be possible to use the test library almost as before without using modules at all, as this is necessary for a certain group of tests, e.g. ones which are strict on the list of available modules. 

The initial modularization proposal is based on the information extracted by 'jdeps' tool. 8211358[10] provides details on proposed modules, their names, packages/classes to include, and reasoning why a separate module is required.

3. Final Goal

After all these improvements are implemented, we will have more organized modularized tests libraries which do not have irritating duplications. Common test libraries classes will be in '/test/lib', test libraries specific for test suites will be in 'lib' directory inside in a test suite. The tests will use these libraries by a few meaningful @library and @build directives, and the test libraries layout will look similar to the following:

test
  |-- lib
  |   |-- test.base
  |   |-- test.compiler
  |   |-- test.jfr
  |   `-- test.whitebox
  |-- hotspot/lib
  |   |-- test.hotspot.jvmti
  |   `-- test.hotspot.ctw
  `-- jdk/lib
      |-- test.jdk.client
      `-- test.jdk.security

4. Summary

Although significant progress has been achieved in this field, there is still a long way ahead of us. We have merged three distinct forks into one common test library available to use in any jtreg test suites. All the existing test suites have been switched to use the classes from this test library. Next steps aim at further improving Open JDK developer experience and make test libraries maintenance easier. 8211290, 8211289, and 8211358 have been filed for each step and contain more details on proposed action plans,  I would really like to encourage interested parties to express their opinion in this email thread and/or in the corresponding RFE. Since the adoption of the test library as well as preventing duplication highly depend on awareness people about existing library classes, I would also appreciate people suggesting to use existing classes when appropriate more often during reviews.

Thanks,
-- Igor

[1] https://bugs.openjdk.java.net/browse/JDK-8007142
[2] http://mail.openjdk.java.net/pipermail/serviceability-dev/2013-February/008215.html
[3] https://bugs.openjdk.java.net/browse/JDK-8157957
[4] http://mail.openjdk.java.net/pipermail/hotspot-dev/2016-August/024198.html
[5] https://bugs.openjdk.java.net/browse/JDK-8075327
[6] https://bugs.openjdk.java.net/browse/JDK-8211171
[7] http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-September/055702.html
[8] https://bugs.openjdk.java.net/browse/JDK-8211290
[9] https://bugs.openjdk.java.net/browse/JDK-8211289
[10] https://bugs.openjdk.java.net/browse/JDK-8211358