RFR (S): 8006413: Add utility classes for writing better multiprocess tests in jtreg

[Kelly O'Hair]

On Jan 17, 2013, at 10:34 AM, Joe Darcy wrote:

> Dmitry,
> 
> On 1/17/2013 9:58 AM, Dmitry Samersoff wrote:
>> Kelly,
>> 
>> We have couple of platform depended points that makes shell tests not
>> reliable:
>> 
>> 1. Getting PID of right process
>> 2. Kill running process
>> 3. Remove remaining locks and locked files/directories (e.g. hsperfdata)
> 
> There are many more shell-specific problems too in terms of differences in what is accepted by the shell we run on different platforms, differences in behavior and location of commands, etc.

Yup, I agree with Joe. It's more than these 3 items.
Shells are easily impacted by their environment. Java code is much more predictable, and we are
delivering Java as a product, maybe we need to write more Java, and if this is hard to do, well,
maybe that's a hint to us that we need to fix something.  The old dog food test. ;^)

> 
>> etc
>> 
>> Java has all these problems as well, so I can't imagine why Java based
>> tests (I speak about multiprocess one only) is more reliable than shell
>> ones.
> 

Yes, Java has issues too, I'm am not denying that. But Java can be debugged, is much more readable and
easier to diagnose. It's a real language, and OUR language, we need to use it.

-kto

> Well, we the JDK group can and has written and shipped cross-platform functionality for such tasks as part of the JDK libraries.  We should feel free to use that work in JDK testing too.
> 
> -Joe
> 
>> 
>>   The simplest way to improve reliability of shell based tests is write
>> different scripts for windows and for other platforms.
>>   On my opinion most of problem with shell scripts is the result of
>> attempt to use unix style programming under windows. (e.g. tlist works
>> much better than ps from MKS or Cygwin)
>> 
>> -Dmitry
>> 
>> 
>> On 2013-01-17 21:14, Kelly O'Hair wrote:
>>> Yes the devil is in shell scripts. Yes it's a pain to program client/server tests with pure Java,
>>> but with pure Java you can share that logic much safer than shell scripts.
>>> It's even more painful to get client/server tests to work reliably in shell scripts, on all platforms.
>>> 
>>> Shell tests have been so problematic for so long that we need to start forcing the issue,
>>> 'no more shell tests' should now be the rule.
>>> Developers need to figure out how to share Java client/server logic between tests and stop adding
>>> more to this shell test nightmare.
>>> 
>>> -kto
>>> 
>>> On Jan 17, 2013, at 9:02 AM, Dmitry Samersoff wrote:
>>> 
>>>> Joe,
>>>> 
>>>> The devil is not in shell scripts.  The devil is in the fact that
>>>> harness (jtreg) doesn't have a process management and doesn't support
>>>> client server tests.
>>>> 
>>>> Shell initially designed to manage processes but java not, so replacing
>>>> shell scripts to java based process builder that exec java from within
>>>> java that exec another java doesn't make things better and just leads to
>>>> the new layer of errors.
>>>> 
>>>> I guess we should invest some time and efforts to jtreg and finaly being
>>>> able to write something like (approximately)
>>>> 
>>>> @run/server/timeout=300 someClass
>>>> @run/client someClass
>>>> 
>>>> and then use
>>>> jtreg.getServerPid(),
>>>> jtreg.shutdownServer()
>>>> 
>>>> etc.
>>>> 
>>>> Then I vote with two hands for getting rid of all shell based tests.
>>>> 
>>>> But today I see no benefits from replacing
>>>> 
>>>> java -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -version |
>>>>  grep "warning: Using incremental CMS is deprecated"
>>>> 
>>>> to a 16 lines of Java code (
>>>> http://cr.openjdk.java.net/~brutisso/8006398/webrev.00/test/gc/startup-warnings/TestCMSIncrementalMode.java.html
>>>> )
>>>> 
>>>> 
>>>> -Dmitry
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 2013-01-17 20:34, Joe Darcy wrote:
>>>>> On 1/17/2013 7:15 AM, Christian Törnqvist wrote:
>>>>>>> What is benefits of this library over usual shell script?
>>>>>> It works on all platforms, also the code is a lot cleaner and easier
>>>>>> to maintain.
>>>>> Shell tests are awful. We should strive to not only stop adding more
>>>>> shell tests, I think we should work to convert existing shell tests into
>>>>> Java programs.
>>>>> 
>>>>> I've been looking a bit at the set of tests with transient failures are
>>>>> shell tests are overrepresented on that list.  Writing cross platform
>>>>> shell code is tricky and the shells on some platforms have a history of
>>>>> being at least one of slow and unreliable. Additionally, the test
>>>>> themselves often have bugs.  For example, Alan Bateman just recently
>>>>> pushed a changest to have the tests properly respect environment
>>>>> variables passed in by jtreg [1].
>>>>> 
>>>>> -Joe
>>>>> 
>>>>> [1] 8005978: shell tests need to use the $COMPILEJDK for javac, jar and
>>>>> other tools
>>>>> http://hg.openjdk.java.net/jdk8/tl/jdk/rev/7da291690aa0
>>>>> 
>>>>>> Thanks,
>>>>>> Christian
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Dmitry Samersoff
>>>>>> Sent: den 17 januari 2013 15:59
>>>>>> To: Christian Törnqvist
>>>>>> Cc: Coleen Phillimore; hotspot-dev at openjdk.java.net
>>>>>> Subject: Re: RFR (S): 8006413: Add utility classes for writing better
>>>>>> multiprocess tests in jtreg
>>>>>> 
>>>>>> On 2013-01-17 18:10, Christian Törnqvist wrote:
>>>>>>> Hi Dmitry,
>>>>>>> 
>>>>>>>> 1. I'm second to Coleen. Is it possible to add more comments
>>>>>>>> explaining how to use this tools.
>>>>>>> My intention is to write a detailed guide to writing jtreg tests,
>>>>>>> including the use of these (and future) utility classes.
>>>>>> Good.
>>>>>> 
>>>>>>>> 2. Is it possible to create your own exception rather than use
>>>>>>>> Exceptioin class directly.
>>>>>>> I've changed it to use RuntimeException based on feedback from other
>>>>>>> people
>>>>>> Please create your own exception class. I would like to be able to
>>>>>> distinguish between exceptions that comes from different parts of tests.
>>>>>> 
>>>>>>>> 3. Who is responsible to kill started process?
>>>>>>> This is really a task for the jtreg framework. Honestly it doesn't
>>>>>>> really do this very well right now, since the process handling in Java
>>>>>>> is not good enough to deal with this. I think this is a something we
>>>>>>> should look into addressing in jtreg rather than having the tests
>>>>>>> worry about it.
>>>>>> What is benefits of this library over usual shell script?
>>>>>> 
>>>>>> -Dmitry
>>>>>> 
>>>>>> 
>>>>>>> Best regards, Christian
>>>>>>> 
>>>>>>> -----Original Message----- From: Dmitry Samersoff Sent: den 17 januari
>>>>>>> 2013 14:14 To: Coleen Phillimore Cc:
>>>>>>> hotspot-dev at openjdk.java.net Subject: Re: RFR (S): 8006413: Add
>>>>>>> utility classes for writing better multiprocess tests in jtreg
>>>>>>> 
>>>>>>> Christian,
>>>>>>> 
>>>>>>> 1. I'm second to Coleen. Is it possible to add more comments
>>>>>>> explaining how to use this tools.
>>>>>>> 
>>>>>>> 
>>>>>>> 2. Is it possible to create your own exception rather than use
>>>>>>> Exceptioin class directly.
>>>>>>> 
>>>>>>> 
>>>>>>> 3. Who is responsible to kill started process?
>>>>>>> 
>>>>>>> -Dmitry
>>>>>>> 
>>>>>>> 
>>>>>>> On 2013-01-16 17:53, Coleen Phillimore wrote:
>>>>>>>> Christian, Can you write up (somewhere!) what these are for and how
>>>>>>>> these are going to be used?  I don't understand just from this
>>>>>>>> code.   Does jtreg know enough to build these classfiles?   Do we
>>>>>>>> have to write shell scripts to refer to these? Thanks, Coleen
>>>>>>>> 
>>>>>>>> On 1/16/2013 7:34 AM, Christian Törnqvist wrote:
>>>>>>>>> Hi everyone,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> This RFE adds a few utility classes to make it a bit easier to write
>>>>>>>>> multi-process tests in jtreg, webrev can be found at
>>>>>>>>> http://cr.openjdk.java.net/~brutisso/8006413/webrev.00/
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> Christian
>>>>>>> -- Dmitry Samersoff Oracle Java development team, Saint Petersburg,
>>>>>>> Russia * Give Rabbit time, and he'll always get the answer
>>>>>>> 
>>>>>> -- 
>>>>>> Dmitry Samersoff
>>>>>> Oracle Java development team, Saint Petersburg, Russia
>>>>>> * Give Rabbit time, and he'll always get the answer
>>>> 
>>>> -- 
>>>> Dmitry Samersoff
>>>> Oracle Java development team, Saint Petersburg, Russia
>>>> * Give Rabbit time, and he'll always get the answer
>> 
>