[rfc] [icedtea-web] blacklist for reproducers

[Jiri Vanek]

On 04/09/2012 12:56 AM, Omair Majid wrote:
> On 04/06/2012 08:56 AM, Jiri Vanek wrote:
>> Hi!
>>
>> This is second from total of 5 patches which are introducing Tets For
>> Applets[1]
>>
>> Thus particular one is introducing  blacklist. The test-engine is
>> delivered to class-path together with all reproducers' testcases. If the
>> file form engine is interface, have no @Test methods or does not meet
>> constructor requirements, then the file fails, and this is making our
>> statistics misleading.
>> Also I'm expecting growing of engine (now I'm planing @bug and @browser
>> annotations and some more logic around. See content of ignore-list after
>> all this changes[2]. Also I would like to refactor ServerAccess class
>> little bit in long term (as it have grown littlebit and have to much
>> inner classes right now) future...
>>
>> Probably cleaner solution will be to separate testengine from testaces
>> more then it is now, but I have tried this refactoring and I was getting
>> deeper and deeper and it was bringing more and more issues. From this
>> I'm really for blacklist rather then for refactoring.
>> Also one more benefit is here - you can easili blacklist all reproducers
>> except one or two fro make ;)
I was  afraid you will say something like this...
>
> I feel rather uncomfortable with this. I really think the underlying
> problem should be fixed instead. Using band-aids like this works in the
> short-term but makes icedtea-web worse in the long term.
>
> If there is some effort required to fix the test-engine, lets do that.
> If the tests are wrong, lets remove them. If it makes sense to run only
> a few tests, adding that would be pretty nice (like how jtreg does it by
> allowing a user to specify a directory to find tests in) but a blacklist
> looks really fishy.
>
> Omair

So lets dig what is necessary to do.
We have

netx directory with sources of nex
├── javax
│   └── jnlp
└── net
     └── sourceforge
         ├── jnlp
...
         │   ├── tools
         │   └── util
...
         └── nanoxml

and we have
tests directory with
├── jnlp_tests  - source files and testcases for reproducers
│   ├── signed
...
│   └── simple
...
├── junit-runner - launcher for both testsuites
├── netx - directory containg  unittests, pac tests, *infrastructure for reproducers* and *tests of 
infrastructure* (virtual server...)
│   ├── jnlp_testsengine - *both* infrastructure for reproducers *and* tests of infrastructure
│   │   └── net
│   │       └── sourceforge
│   │           └── jnlp
│   ├── pac - sources of pac tests
│   └── unit - sources of unittests
│       └── net
│           └── sourceforge
│               └── jnlp
│                   ├── application
│                   ├── runtime
│                   ├── templates
│                   └── util
│                       └── replacements


The we have compile-time and run-time dependencies in makefile (I'm usinf sources locations instead 
of tests.build in this text wilfully)
They are currently :
compile-time of unittests:
netx and tests/netx/unit
run-time of unittests:
junit-runner netx and tests/netx/unit


compile-time of reproducer's sources:
netx  and tests/jnlp_tests/signed|simple/reproducer/src/
compile-time of reproducer's testcases:
netx tests/netx/jnlp_testsengine  and tests/jnlp_tests/signed|simple/reproducer/testcase/ (all 
testcases together)

runtime-time of reproducer's sources:
junit-runner  netx  and from its jnlp/html (mostly in  jnlp_tests_server with jars created from 
tests/jnlp_tests/signed|simple/sources/testcase/)
run-time of reproducer's testcases:
junit-runner netx tests/netx/jnlp_testsengine  and 
tests/jnlp_tests/signed|simple/reproducer/testcase/(all testcases together)


And second thing is where the classes of tests are taken:

for unittests all files inisde tests/unit
for reproducers all files tests/netx/jnlp_testsengine and 
tests/jnlp_tests/signed|simple/reproducer/testcase/

The error is obvious. tests/netx/jnlp_testsengine contains both tests and logic

My opinion here will be to unify above as much as possible.
When you will check [1] for browserTests-BugAnnotation [2] you will see how few other issues "solved 
below" raised.


[1]http://mail.openjdk.java.net/pipermail/distro-pkg-dev/2012-March/017799.html
[2]http://mail.openjdk.java.net/pipermail/distro-pkg-dev/attachments/20120320/8ef6bdfe/browserTests-BugAnnotation-0001.diff


1) move all tests from tests/netx/jnlp_testsengine to tests/netx/unit where they belongs IMHO
1.5) With this refactoring I would like to extract all inner classes from ServerAccess
2) fix compile and runtime class-path of unittests for tests/netx/jnlp_testsengine
3) during reproducers run launch *just* compiled testcases
4) no more tests in tests/netx/jnlp_testsengine, framework will be tested in unittests (same as (1) 
but just for ensuring..:) )
4.5) unittets and reprodyucers runs should have same classapth in runtime/compiletime. => unify them 
in some variable(s)(?)
5) I'm also for adding tests/netx/jnlp_testsengine to compile class-path of junit-runner (as it 
looks useful from [1,2] point of view.
6) Maybe after all of above done the jnlp_testsengine can be renamed and moved somewhere else. (and 
also jnlp_tests probably should be moved to tests/netx/) But This should be done as separate changeset.

1-6 - ServerAccess api should not be changed, although maybe some methods private/public/protected 
statuses will be transformed due to moving of tests)


What do you think?


For launching just some tests I would like then introduce some whitelist. The logic will be same as 
was introduced in  this discarded  patch, but default value will be ".*"  Before wanted run it will 
be possible to enter regex matching the set of reproducers determined for run.


What do you think?

J.