jtreg testing integrated

[Joseph D. Darcy]

Hello Andrew.

Andrew John Hughes wrote:
> On 21/05/2008, Joseph D. Darcy <Joe.Darcy at sun.com> wrote:
>> Hello Mark.
>>
>>  Mark Wielaard wrote:
>>
>>> Hi Joe,
>>>
>>> On Mon, 2008-05-19 at 09:59 -0700, Joseph D. Darcy wrote:
>>>
>>>
>>>> Mark Wielaard wrote:
>>>>
>>>>
>>>>> On Mon, 2008-05-19 at 11:21 +0200, Mark Wielaard wrote:

[snip]

>>  Those may take a little while to figure out a good fix in the sense of
>> still testing the networking code outside Sun's firewall as opposed to just
>> adding "if(!insideSun()) pass" logic.  I'll talk to the networking team
>> about this.
>>
> 
> Well I don't believe anyone was suggesting such a simplistic hack.  We
> need a well-known external host with good uptime that would handle
> such requests without breaking a sweat.

Or at least information about what kinds of services are expected so 
they could be replicated elsewhere.

>>  On a related note, there are times where it could be useful for a test to
>> have somewhat different behavior on Sun's production JDK versus OpenJDK.
>> For example, I'm the author of the
>> java/lang/reflect/Generics/Probe.java test which fails on
>> OpenJDK since it references some production-only classes.  There are a few
>> options to fix this.  One way would be to remove all reference to the
>> production-only classes, which would reduce the usefulness of the test for
>> the production code; conversely, the whole test could be moved to closed,
>> reducing the test coverage of OpenJDK.  The test could be split into open
>> and closed parts, complicating maintenance.  So instead I changed the test
>> to only look at the production classes for a production build.  I test for a
>> production build using
>>
>>  System.getProperty("java.runtime.name").startsWith("Java(TM)")
>>
>>  Out of the box, our OpenJDK builds have "java.runtime.name" start with
>> "OpenJDK"; is that true in your IcedTea/OpenJDK builds too?
>>
> 
> I assume what you mean by a 'production build' is the code being used
> to build a proprietary JDK with additional non-Free classes.  If this

Yes, by "production build" I mean a build of Sun's product binaries with 
the non-Free, non-open bits.

> is the case, I would hope that, in the long term, these will no longer
> be needed and the 'production build' will be just a build from the
> GPLed codebase like any other.

That would certainly make the engineering easier :-)

>  I mention this because I think the
> term could be a little disparaging to outsiders, as it implies only a
> build with proprietary Sun code is considered production ready.

Yes, getting accurate, descriptive, and neutral (and concise) wording 
can be tricky, as "proprietary" is also disparaging in some contexts ;-)

> I wouldn't add any code to rely on the name of a build from the GPLed
> code.  It's just asking for trouble further down the line.  Besides,
> by including such tests, you're effectively wrongly favouring some
> GPLed distributions over others.

Well, the tests in question come with the code, both under GPL, so if 
someone wanted to have java.runtime.name=Fred for some reason, the tests 
in question could be updated to expect "Fred" instead.

> Is a test for the production system (with an appropriate comment so we
> know that's what it is) not sufficient?

That is probably sufficient for the cases in question.

There are really three cases of possible interest: Sun's traditional 
(proprietary) product bits, a "usual" OpenJDK build, and something else. 
  A "usual" OpenJDK build perhaps is something yet to come about; I'm 
thinking there may be conventions we all choose to follow as good 
practices that may affect how the tests are written in some way.  A 
"something else" build would, say, be based on the OpenJDK code, but 
choose to go against usual practice in some manner.

The cases currently under consideration just need to make a decision 
based on "Sun's traditional (proprietary) product" versus "not Sun's 
traditional (proprietary) product."

>>  For the javax/script tests, I was going to add "if (no javascript engine
>> and production JDK) fail else note/warning" logic so that our production
>> build gets the same coverage and if an OpenJDK build come with a javascript
>> engine, it would get testing too.
>>
> 
> Wouldn't testing the result of
> ScriptEngineManager.getEngineFactories() suffice? Why do you need to
> specifically test for a production build?

In Sun's proprietary product JDK we've chosen to ship a javascript 
engine; this is *not* required by the specification.  The OpenJDK 
sources do not include a javascript engine; although people could add 
one of course.  I'd like the tests to require javascript to be present 
in Sun's proprietary product JDK; otherwise, if a javascript engine 
happens to be present, it should be tested the same way, but not having 
the engine shouldn't be fatal.  This would preserve the same degree of 
testing of the product bits, allow the tests to pass out of the box for 
an OpenJDK build, and perform the same tests for an OpenJDK  build if an 
engine happens to be present.

-Joe