RFR 8181299/10, Several jdk tests fail with java.lang.NoClassDefFoundError: jdk/test/lib/process/StreamPumper

[David Holmes]

On 3/06/2017 5:10 AM, Igor Ignatyev wrote:
> 
>> On Jun 2, 2017, at 9:14 AM, Ioi Lam <ioi.lam at oracle.com> wrote:
>>
>> On 6/2/17 8:44 AM, Ioi Lam wrote:
>>>
>>> On 6/2/17 6:40 AM, Chris Hegarty wrote:
>>>> On 02/06/17 00:14, Ioi Lam wrote:
>>>>> ...
>>>>>
>>>>> The gem is hidden in the compile.0.jta file. It contains something like:
>>>>>
>>>>>       -sourcepath <blahblah>:/jdk/foobar/test/lib:<blahblah>
>>>>>
>>>>> So if my test refers to a class under /test/lib, such as
>>>>> jdk.test.lib.process.ProcessTools, javac will be able to locate it under
>>>>> /jdk/foobar/test/lib/jdk/test/lib/process/ProcessTools.java, and will
>>>>> build it automatically.
>>>>>
>>>>> So really, there's no reason why the test must explicitly do an @build
>>>>> of the library classes that it uses.
>>>>
>>>> Sure, you're relying on the implicit compilation of dependencies
>>>> by javac. Look at the output, where it compiles the library
>>>> classes to.  It is part of the classes directory for the
>>>> individual test. That means that the library classes will need
>>>> to be compiled many many times.  The @build tag will compile
>>>> the library classes to a common output directory, where they
>>>> can be reused ( unless I'm missing something ).
>>>>
>>>> -Chris.
>>> Yes, @build will compile classes so that they can be reused. But why should it be the responsibility of every test to do this?
>>>
>>> To reuse my malloc metaphore -- is it reasonable for every program that uses malloc to explicitly build libc?
>>>
>>> By the way, jtreg arranges the output directory of the test by the directory they sit in, so
>>>
>>>   jdk/test/foo/bar/XTest.java
>>>   jdk/test/foo/bar/YTest.java
>>>
>>> will all output their .class files to the same directory. Therefore, the amount of duplicated classes is not as bad as you might think. We've been omitting the @build tags in the hotspot tests and we haven't seen any problems.
>>>
>>> - Ioi
>> To avoid repeat compilation of the library classes, a more reasonable solution would be:
>>
>> [1] Before test execution -- scan all the selected test to find all libraries specified by @library tags
>>
>> [2] Fully compile all the libraries into their own output directories
>>
>> [3] Then, start execution of the selected tests
> 
> unfortunately, it is not that simple, there are at least 2 problems w/ that approach:
>   1. some of library classes have extra module dependency, e.g. jdk.test.lib.management.* depend on jdk.management module, ExtendedRobot (from jdk/test/testlibrary) depends on java.desktop. so compiling the whole library will require extra module dependency, which might be unneeded for the selected tests, as a result we won't be able to run these tests on configurations w/ limited module set.
>   2. to make our tests packagefull, we had to add '@library /' to many hotspot/test/compiler tests, so we will have to compile all files from hotspot/test.
> 
> my take on all of this is that determination of output directory for classes is buggy, it uses directory of a @build or @run target to decide where put all produced classes files, but it should have mapping between source and destination paths instead, so all classes from jdk/test/foo/bar/ will go to a test scratch directory and all classes from /test/lib/ (assuming they are declared as @library) and /jdk/test/lib/ to different common directories which will be later added to classpath for the tests which use these libraries.

But unless you explicitly compile the library classes you can't control 
where the class files are placed. The tests are compiled with a "-d" 
directive, so all classes, directly and implicitly compiled will be 
relative to that directory based on their package. If every test were 
declared in a package based on the source arrangement then jtreg would 
be able to use a common output directory.

Neither suggested approach seems a great solution to me. Implicit 
compilation wastes effort rebuilding the libraries. Explicit compilation 
is instrusive and difficult to get right - and I have no idea how to get 
the module dependency stuff sorted out.

Maybe the design flaw here is attempting to combine a test library with 
the tests that use it. You either want it to be a binary library jtreg 
can be pointed at, or you need a way to tell jtreg to build the library 
first and then use. But IIUC jtreg isn't set up to handle that - but it 
could be handled by running jtreg via make.

My 2c.

David

> Thanks,
> -- Igor
> 
>   
>