RFR (S): 8006413: Add utility classes for writing better multiprocess tests in jtreg

[Stuart Marks]

Wow, a huge thread here. A few general comments.

Shell Tests.

While shell scripts seem to make it really easy to manage subprocesses and 
create pipelines, in practice we've found that they really have caused a lot of 
problems. Joe mentioned that they seem to contribute more than their proportion 
of failures during pre-integration testing. I've also run across several 
instances of "bugs" in shell tests, where they mishandle unexpected conditions 
or destroy valuable diagnostic information.

In the JDK regression suite we have a lot of intermittent failures. To the 
extent that shell tests contribute more than their share of such failures, 
we're strongly considering rewriting these tests in Java in order to make them 
more reliable. And, as already has been stated, we'd like to discourage writing 
of new shell tests.

Process Infrastructure in JTreg.

If we're going to be writing tests in Java that manage subprocesses, this will 
require some support and infrastructure code. Shouldn't this be put into a 
place like JTreg where everybody can use it?

Perhaps conceptually yes. The main problem is that JTreg evolves separately and 
asynchronously from the platform. If a test needed a new bit of infrastructure, 
it would have to wait until it was added to JTreg. If the infrastructure 
library is in the same repo as the tests, they could all be updated simultaneously.

A Single Shared Infrastructure Library.

If we're to have a library, shouldn't we have a single library that everybody 
uses? Again, conceptually yes, but there are a couple layers of problems in the 
way.

The first is that hotspot and jdk are different repositories, and trying to 
have a single one would introduce more dependencies across the repositories, 
which we all understand is very difficult to manage. Keeping the libraries in 
synch, using some external process (e.g. a script run regularly) would still be 
a problem. If a change in, say, the jdk version of the test library would cause 
a problem in the hotspot tests, that would be a big effort to reconcile.

This occurs even if there's a test library that's shared among a bunch of tests 
in the same repo. That library has to accommodate everybody's requirements, and 
it can't be changed unless it's assured that it doesn't break anything. 
Maintaining and evolving such a library will require a significant amount of 
effort. Creating such a library is a good idea, but we're not there yet. I 
don't think our test library code is mature enough yet.

What I don't want to have happen is for there to be a "test library" that's 
simply a dumping ground for code that is potentially sharable, that might be 
useful across a bunch of tests, but which a) nobody actually uses, because they 
don't want to be exposed to breakage if it's changed, and b) which nobody 
maintains, because they're afraid of changing something that might break somebody.

We're left with the situation of having each area of the code define its own 
library, like the RMI test library is used only by RMI. This means that some 
test library code will be duplicated. This is bad, but it's less bad than what 
I outlined above, and I hope it's only temporary.

s'marks