RFR (S): 8006413: Add utility classes for writing better multiprocess tests in jtreg

[Jonathan Gibbons]

That does not sound like a viable solution to me.   We should not have 
test code in the test harness -- we will have too much trouble 
synchronizing the right version of the harness with the tests.

We need a greater separation between the harness and the tests.

-- Jon

On 01/17/2013 09:28 AM, Dmitry Samersoff wrote:
> Jonathan,
>
> Just to clarify, I propose:
>
> 1. Add TestTools directory to JTREG workspace.
> 2. Build TestTools.jar together with jtreg
> 3. Maintain and distribute TestTools.jar together with
>     jtreg under jtreg policies.
> 4. Add <jtreg distribution>/TestTools.jar to default classpath for
>     all tests.
> 5. Use import com.oracle.jtreg.tools.* when necessary.
>
> -Dmitry
>
>
>
> On 2013-01-17 21:23, Jonathan Gibbons wrote:
>> I think there's a confusion here between an external dependency on the
>> test harness (which has generally been deemed acceptable) and a
>> potentially new external dependencies on test libraries in other
>> repositories in the forest, which would see less acceptable to me.
>>
>> There /are/ times when cut-n-paste is the correct solution. :-)
>>
>> -- Jon
>>
>> On 01/17/2013 09:17 AM, Kelly O'Hair wrote:
>>> The C compiler, bash, libraries, unix utilities, aren't in the repo.
>>> We will have and always have
>>> external dependencies. I agree we need to limit them, but a testing
>>> harness (ideally one) is a very
>>> valid external dependency to have.
>>>
>>> Hotspot needs to get lined up with the standard jdk testing, jtreg is
>>> already used in  jdk, JPRT,
>>> QE teams, and many developers are very familiar with it.
>>>
>>> -kto
>>>
>>> On Jan 17, 2013, at 6:10 AM, Christian Törnqvist wrote:
>>>
>>>> Well, jtreg is not in the hotspot repo (and shouldn't be) and
>>>> suddenly you end up having an external dependency. I'd like to keep
>>>> the tests free from dependence on jtreg (along with other external
>>>> resources).
>>>>
>>>> Thanks,
>>>> Christian
>>>>
>>>> -----Original Message-----
>>>> From: Dmitry Samersoff
>>>> Sent: den 17 januari 2013 15:06
>>>> To: Christian Törnqvist
>>>> Cc: Stuart Marks; hotspot-dev at openjdk.java.net; Jonathan Gibbons
>>>> Subject: Re: RFR (S): 8006413: Add utility classes for writing better
>>>> multiprocess tests in jtreg
>>>>
>>>> Christian,
>>>>
>>>> On my opinion something like
>>>>
>>>> import com.oracle.jtreg.tools.*
>>>>
>>>> is less painful than maintain several independent implementations of
>>>> the same ideas.
>>>>
>>>> -Dmitry
>>>>
>>>>
>>>> On 2013-01-17 18:01, Christian Törnqvist wrote:
>>>>>> Is it possible to have a library like this one included into jtreg
>>>>>> and available for everybody ?
>>>>> The problem with this is that suddenly the tests have a dependency on
>>>>> jtreg. One (of many) nice things with the jtreg tests is that they can
>>>>> be run as a normal java program without the need to involve jtreg,
>>>>> making debugging and troubleshooting them a lot easier.
>>>>>
>>>>> Best regards, Christian
>>>>>
>>>>> -----Original Message----- From: Dmitry Samersoff Sent: den 17 januari
>>>>> 2013 14:20 To: Stuart Marks Cc: Christian Törnqvist;
>>>>> hotspot-dev at openjdk.java.net; Jonathan Gibbons Subject: Re: RFR (S):
>>>>> 8006413: Add utility classes for writing better multiprocess tests in
>>>>> jtreg
>>>>>
>>>>> Stuart,
>>>>>
>>>>>> I work on the JDK core libraries. It turns out that there is a
>>>>>> similar library used for testing RMI.
>>>>> I guess if we take close look to exising tests we would find couple
>>>>> more copies of the same ideas.
>>>>>
>>>>> Is it possible to have a library like this one included into jtreg and
>>>>> available for everybody ?
>>>>>
>>>>> -Dmitry
>>>>>
>>>>>
>>>>> On 2013-01-17 01:54, Stuart Marks wrote:
>>>>>> Hi everybody,
>>>>>>
>>>>>> I work on the JDK core libraries. It turns out that there is a
>>>>>> similar library used for testing RMI. Take a look at
>>>>>> jdk/test/java/rmi/testlibrary. It's been around for quite some time
>>>>>> already, but I've been doing some maintenance on it recently.
>>>>>> (Also, it still needs more work.)
>>>>>>
>>>>>> Since the RMI library is in the jdk repo and these test utilities are
>>>>>> in hotspot, it seems like sharing of code might be difficult.
>>>>>> But we might want to compare notes. Here are some observations and
>>>>>> comments (though not an actual code review).
>>>>>>
>>>>>> * The use of a pair of threads (StreamPumper) to collect output from
>>>>>> the subprocess is very similar to what RMI does. See the StreamPipe
>>>>>> class in RMI's test library, which are used by the JavaVM utility
>>>>>> class. One thing I did do recently was to make sure that the
>>>>>> subprocess is waited for and the threads are joined. This hadn't been
>>>>>> done before, leading to race conditions where some subprocess output
>>>>>> was dropped.
>>>>>>
>>>>>> * Since these threads are specifically joined, I'm not sure of the
>>>>>> utility of making them daemon threads.
>>>>>>
>>>>>> * My style in writing test code and test library code is to be quite
>>>>>> verbose about everything and to check every possible error condition.
>>>>>> Nothing is worse than seeing a test fail and not having enough
>>>>>> information in the log to find out why. Actually, I can think of
>>>>>> something worse: having a test pass erroneously because some error
>>>>>> condition wasn't checked. (Yes, this has happened.)
>>>>>>
>>>>>> * Another style point is that I think it's OK for test library code
>>>>>> just to throw exceptions out to the caller. My test code usually has
>>>>>> 'throws IOException' or even 'throws Exception' on most of the
>>>>>> methods, including main, so having test library code throw checked
>>>>>> exceptions is reasonable. Jtreg will catch exceptions thrown from
>>>>>> main and log them. Usually you want tests to fail quickly at the
>>>>>> point of error. So, I concur with Vitaly's concern about ProcessTools
>>>>>> handling of IOException and about letting interrupts propagate.
>>>>>>
>>>>>> * Actually I have a suggestion from Mandy Chung for RMI's StreamPipe
>>>>>> threads to catch exceptions and report them back to their "parent"
>>>>>> JavaVM. This is probably a rare occurrence but it does ensure that
>>>>>> any errors that occur here won't go unnoticed. See
>>>>>> JDK-8005827 (
>>>>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8005827 )
>>>>>>
>>>>>> * Regarding jtreg and test library code, it's pretty simple to use.
>>>>>> In each test that uses the library code, one adds the @library tag to
>>>>>> declare the library. It's probably a good idea to add an @build tag
>>>>>> to build each library class that's used in the test. Things usually
>>>>>> work in the absence of @build tags, because of implicit compilation,
>>>>>> but it's kind of fragile.
>>>>>>
>>>>>> Anyway, I just wanted to share some notes and ideas.
>>>>>>
>>>>>> s'marks
>>>>>>
>>>>>>
>>>>>> On 1/16/13 4:34 AM, Christian Törnqvist wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> ?
>>>>>>>
>>>>>>> This RFE adds a few utility classes to make it a bit easier to write
>>>>>>> multi-process tests in jtreg, webrev can be found at
>>>>>>> http://cr.openjdk.java.net/~brutisso/8006413/webrev.00/
>>>>>>>
>>>>>>> ?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Christian
>>>>>>>
>>>>> -- Dmitry Samersoff Oracle Java development team, Saint Petersburg,
>>>>> Russia * Give Rabbit time, and he'll always get the answer
>>>>>
>>>> -- 
>>>> Dmitry Samersoff
>>>> Oracle Java development team, Saint Petersburg, Russia
>>>> * Give Rabbit time, and he'll always get the answer
>