Fwd: Re: type anno test code check in

[Jonathan Gibbons]

I think you misunderstood my comment.

I am suggesting, as a reviewer and someone who might one day have to 
debug these tests, that you should dump source code into a file with a 
.java extension. It is important for a well written test that the 
artifacts for failing tests should be clearly identified, and whenever 
possible, should be in a form that is usable outside the context of the 
test.  For example, if you have a test case involving one or more 
classes that fail, I would expect to see a clearly identified directory 
containing source files that I could pass to javac to reproduce and 
thence debug the failure.

-- Jon

On 09/16/2013 05:34 PM, Charlie Wang wrote:
> Hi Jonathan,
>   Thanks for the idea, but I don't think so. In order to identify 
> source code of different tests, I used a prefix(the test class's name) 
> before _source.txt. e.g. 
> ConstructorGetAnnotatedReturnTypeTest_source.txt. But none of the 
> classes in the source code have such a name(they are all like 
> TypeAnnoCls1). Besides, this is just for users' viewing. I think .txt 
> files should be OK.
>
>
>
>
>
> - Charlie
>
>
> On 2013/9/16 23:03, Jonathan Gibbons wrote:
>> Charlie,
>>
>> If it's source code, why would you not dump it into a .java file?
>>
>> -- Jon
>>
>>
>> On 09/16/2013 12:33 AM, Charlie Wang wrote:
>>> Hi Steve,
>>>   Made the update according to your comments. The test will dump the 
>>> source code into a .txt file if it fails. Please review the 
>>> attachment. I will update webrev later.
>>>
>>>
>>> - Charlie
>>>
>>> On 2013/9/13 1:12, Steve Sides wrote:
>>>> On 9/12/2013 4:05 AM, Charlie Wang wrote:
>>>>> Hi Joel,
>>>>>   I ran the test on windows64, linux64, solaris32, mac64, and only 
>>>>> found one failed test - ClassGetAnnotatedInterfacesTest. I think 
>>>>> it is due to code change to Class.getAnnotatedInterfaces(). In the 
>>>>> builds a while before b106 it didn't return null with primitive 
>>>>> type class input, but now it does. I didn't find any bug to track 
>>>>> this change when I searched with label "type-annotations" or 
>>>>> "annotations".
>>>>>   Besides that, I've made a little update to the comments of every 
>>>>> tests explaining how the test code is constructed and run.
>>>>>   By setting Helper.DEBUG = true/false, it displays/hides the test 
>>>>> source code on the command line output, which is constructed for 
>>>>> the test. I hope this doesn't seem overwhelmingly friendly.
>>>> Can you make it so is displays or leave on disk the test source for 
>>>> failed test cases by default apart from DEBUG?
>>>> Since we're really most interested in failed test cases and the 
>>>> ability to easily reproduce them with a small(er) test case, that 
>>>> would be the desired behavior.
>>>> I.E., we don't want to say, "here's  bug, to reproduce, run the 
>>>> whole test", or even "you can set DEBUG and go fish for the failed 
>>>> test case from the thousands in the debug output"....put in the 
>>>> most unfriendly terms. :)  (I do smile when I write these things).
>>>>
>>>> -steve
>>>>
>>>>>   As suggested by Alex, the interfaces are introduced for 
>>>>> extension. DeclarationGenerator is an interface for small code 
>>>>> declarations. TestCaseGenerator is an interface for code 
>>>>> generation of test cases. Helper constructs declarations and 
>>>>> compile source code (better explanations are in Helper's 
>>>>> comments). TestCodeGenerator is a utility class for test case 
>>>>> generation.
>>>>>
>>>>> Attached are the updated test files. I will update the webrev link 
>>>>> later.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> - Charlie
>>>>>
>>>>> On 2013/9/11 22:44, Joel Borggren-Franck wrote:
>>>>>> Hi Charlie,
>>>>>>
>>>>>> These tests should go into TL, I can commit them there. The test 
>>>>>> review
>>>>>> cycle should be on core-libs-dev at openjdk.java.net.
>>>>>>
>>>>>> However the test are not ready yet.
>>>>>>
>>>>>> If you run the tests vs jdk8-b106 or a recent copy of TL there is 
>>>>>> a few
>>>>>> failures. How can I as a developer see the exact source which 
>>>>>> caused the
>>>>>> failure?
>>>>>>
>>>>>> Further, there is an @author tag between two jtreg tags, @build and
>>>>>> @run in almost every file. Please don't do that.
>>>>>>
>>>>>> As a user of these tests there needs to be a way for me to:
>>>>>>
>>>>>> 1) Understand the tests / ensure that the tests are correct
>>>>>> 2) Run the tests
>>>>>> 3) Understand what causes tests failures
>>>>>>
>>>>>> Out of these only 2) is currently easy. 1) is hard and 3) looks
>>>>>> impossible.
>>>>>>
>>>>>> Here is what you need to do to bring this forward:
>>>>>>
>>>>>> - At a minimum you need to document how the tests work. It is not 
>>>>>> clear
>>>>>>    at the moment.
>>>>>>
>>>>>> - Please explain why are the helpers split out into 2 interfaces 
>>>>>> and 4
>>>>>>    classes?
>>>>>>
>>>>>> - You also need to improve failure friendliness. If a tests fail, it
>>>>>>    must be possible to see the java source of the generated 
>>>>>> program that
>>>>>>    fails together with a decent explanation of what went wrong.
>>>>>>
>>>>>> If this is currently available, please indicate how to do it.
>>>>>>
>>>>>> cheers
>>>>>> /Joel
>>>>>>
>>>>>> On 2013-09-11, Charlie Wang wrote:
>>>>>>> Hi,
>>>>>>>    I'm looking for a committer to help me check in type annotation
>>>>>>> reflection test code. Could someone give me a hand?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Charlie
>>>>>>>
>>>>>>> -------- Original Message --------
>>>>>>> Subject:     Re: type anno test code check in
>>>>>>> Date:     Tue, 10 Sep 2013 10:29:02 -0700
>>>>>>> From:     Alex Buckley <alex.buckley at oracle.com>
>>>>>>> Organization:     Oracle Corporation
>>>>>>> To:     Charlie Wang <charlie.wang at oracle.com>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ask on type-annotations-dev for a Committer in the Type Annotations
>>>>>>> Project to push the tests for you. The tests will go in the jdk 
>>>>>>> repo.
>>>>>>>
>>>>>>> Alex
>>>>>>>
>>>>>
>>>>
>>
>