Fwd: Re: type anno test code check in

[Jonathan Gibbons]

Charlie,

As a general warning, note that jtreg limits the amount of output that 
is recorded from the console log.   If you exceed the limit, jtreg will 
preserve the begnning of the output (to record start up messages) and 
the end of the output (to record summary messages, etc) and will omit 
text in between. Therefore, you may not want to record huge amounts of 
unnecessary output to the console log. Saving info to files in the 
scratch directory is fine.

I endorse Steve's suggestion to make the default behavior be enough for 
initial debugging of failed test cases.  This is good as a general 
practise for all tests, especially when it may be difficult to recreate 
the failure for environmental reasons.

-- Jon

On 09/12/2013 10:12 AM, Steve Sides wrote:
> On 9/12/2013 4:05 AM, Charlie Wang wrote:
>> Hi Joel,
>>   I ran the test on windows64, linux64, solaris32, mac64, and only 
>> found one failed test - ClassGetAnnotatedInterfacesTest. I think it 
>> is due to code change to Class.getAnnotatedInterfaces(). In the 
>> builds a while before b106 it didn't return null with primitive type 
>> class input, but now it does. I didn't find any bug to track this 
>> change when I searched with label "type-annotations" or "annotations".
>>   Besides that, I've made a little update to the comments of every 
>> tests explaining how the test code is constructed and run.
>>   By setting Helper.DEBUG = true/false, it displays/hides the test 
>> source code on the command line output, which is constructed for the 
>> test. I hope this doesn't seem overwhelmingly friendly.
> Can you make it so is displays or leave on disk the test source for 
> failed test cases by default apart from DEBUG?
> Since we're really most interested in failed test cases and the 
> ability to easily reproduce them with a small(er) test case, that 
> would be the desired behavior.
> I.E., we don't want to say, "here's  bug, to reproduce, run the whole 
> test", or even "you can set DEBUG and go fish for the failed test case 
> from the thousands in the debug output"....put in the most unfriendly 
> terms. :)  (I do smile when I write these things).
>
> -steve
>
>>   As suggested by Alex, the interfaces are introduced for extension. 
>> DeclarationGenerator is an interface for small code declarations. 
>> TestCaseGenerator is an interface for code generation of test cases. 
>> Helper constructs declarations and compile source code (better 
>> explanations are in Helper's comments). TestCodeGenerator is a 
>> utility class for test case generation.
>>
>> Attached are the updated test files. I will update the webrev link 
>> later.
>>
>>
>>
>>
>>
>>
>> - Charlie
>>
>> On 2013/9/11 22:44, Joel Borggren-Franck wrote:
>>> Hi Charlie,
>>>
>>> These tests should go into TL, I can commit them there. The test review
>>> cycle should be on core-libs-dev at openjdk.java.net.
>>>
>>> However the test are not ready yet.
>>>
>>> If you run the tests vs jdk8-b106 or a recent copy of TL there is a few
>>> failures. How can I as a developer see the exact source which caused 
>>> the
>>> failure?
>>>
>>> Further, there is an @author tag between two jtreg tags, @build and
>>> @run in almost every file. Please don't do that.
>>>
>>> As a user of these tests there needs to be a way for me to:
>>>
>>> 1) Understand the tests / ensure that the tests are correct
>>> 2) Run the tests
>>> 3) Understand what causes tests failures
>>>
>>> Out of these only 2) is currently easy. 1) is hard and 3) looks
>>> impossible.
>>>
>>> Here is what you need to do to bring this forward:
>>>
>>> - At a minimum you need to document how the tests work. It is not clear
>>>    at the moment.
>>>
>>> - Please explain why are the helpers split out into 2 interfaces and 4
>>>    classes?
>>>
>>> - You also need to improve failure friendliness. If a tests fail, it
>>>    must be possible to see the java source of the generated program 
>>> that
>>>    fails together with a decent explanation of what went wrong.
>>>
>>> If this is currently available, please indicate how to do it.
>>>
>>> cheers
>>> /Joel
>>>
>>> On 2013-09-11, Charlie Wang wrote:
>>>> Hi,
>>>>    I'm looking for a committer to help me check in type annotation
>>>> reflection test code. Could someone give me a hand?
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Charlie
>>>>
>>>> -------- Original Message --------
>>>> Subject:     Re: type anno test code check in
>>>> Date:     Tue, 10 Sep 2013 10:29:02 -0700
>>>> From:     Alex Buckley <alex.buckley at oracle.com>
>>>> Organization:     Oracle Corporation
>>>> To:     Charlie Wang <charlie.wang at oracle.com>
>>>>
>>>>
>>>>
>>>> Ask on type-annotations-dev for a Committer in the Type Annotations
>>>> Project to push the tests for you. The tests will go in the jdk repo.
>>>>
>>>> Alex
>>>>
>>
>