Regression test failures - could not find entries in JDK Bug System

Tue Oct 4 12:32:47 UTC 2016

Hi Vitaly,

Comments inline.

On 04/10/2016 13:00, Vitaly Provodin wrote:
> Hi Rory,
>
> Thanks a lot for the detailed answer - now it is clear for EAB runs. 
> Sorry I overlooked the note 2.
> Actually, the reason of my mail was to understand that the failures I 
> got (including those two) are expected and can be explained by 
> existing bugs. Totally there were about 300 test failures of java/awt 
> and javax/swing tests in my environment. Most of them are explained by 
> existing bugs.
> For others (about 70) no bugs were found - most probably they are 
> somehow mentioned in Bug System but i could find no entries.
>
> Some of these 70 failures can be caused by specifics in my 
> environment. But others like 
> /javax/swing/JPopupMenu/6800513/bug6800513.java/ may be considered as 
> an issue in Java.
This could be  - JDK-7184956
> The main question here how can I report these issues (if I have no 
> write-access to the Bug System)?
> It looks like time-consuming job doing it via emails...
>

You can report issues via http://bugs.java.com/

> What can you recommend me?
>
> Thanks in advance,
> Vitaly
>
> P.S. now I am able to open number-links via 
> http://download.java.net/openjdk/testresults/9/testresults.html
> But they can not be opened if starting page is 
> http://download.java.net/openjdk/testresults/9
Thanks for that, I will get it looked at.

Rgds,Rory

>
>> On 04 Oct 2016, at 17:02, Rory O'Donnell <rory.odonnell at oracle.com 
>> <mailto:rory.odonnell at oracle.com>> wrote:
>>
>> Hi Vitaly,
>>
>> A little bit of background, the purpose of the posting of EA build 
>> results is to provide consistent, reproducible
>> results that people can generate and compare against in their own 
>> environment.
>>
>> To this end, we do not run the awt, swing and 2d tests, as there are 
>> so many variables and the results
>> can be inconsistent.
>>
>> On 04/10/2016 03:35, Vitaly Provodin wrote:
>>>
>>> Hello,
>>>
>>> My question relates to failures of the Regression tests (jdk/test) 
>>> on OpenJDK8 and how to identify whether a failure is expected or not.
>>>
>>> Running java/awt and javax/swing tests against OpenJDK8 on various 
>>> platforms I got some failures. Trying to find any explanations for 
>>> these failures I looked through the JDK Bug System and could not 
>>> find any bugs explaining them, for example, for the following two tests
>>>
>> As I mentioned above, we don't run these tests as part of the regular 
>> testing see NOTE 2
>> http://download.java.net/openjdk/testresults/9/testresults.html
>> 2) ** Results do not includes test results for awt, swing and 2d.
>>>
>>> /-java/awt/Frame/UnfocusableMaximizedFrameResizablity/UnfocusableMaximizedFrameResizablity.java/
>>>
>>>     Failed.
>>>     Program
>>>     `/Users/builduser/buildAgent/work/13ab0d4c51b0de0b/jbsdk/jdk/Contents/Home/bin/java'
>>>     timed out (timeout set to 240000ms, elapsed time was 240786ms).
>>>
>>>
>>> /- javax/swing/JPopupMenu/6800513/bug6800513.java/
>>>
>>>
>>>     Failed.
>>>     Execution failed: `main' threw exception: java.lang.Exception:
>>>     popup class is: javax.swing.PopupFactory$HeavyWeightPopup,
>>>     expected: javax.swing.PopupFactory$LightWeightPopup
>>>
>>>
>>> I found out that these tests were excluded from runs against JDK 8 
>>> Update Releases Early Access Build (EAB) - see Test Results page at 
>>> http://download.java.net/openjdk/testresults/8. Please note, I do 
>>> not know the status of these results, whether they are actual for 
>>> OpenJDK8 final release or not. Also, I paid attention to that most 
>>> of the failures I am talking about are  not included into 
>>> jdk/test/ProblemList.txt as for now.
>>>
>> As I mentioned above, we don't run these tests as part of the regular 
>> testing see NOTE 2
>> http://download.java.net/openjdk/testresults/9/testresults.html
>> 2) ** Results do not includes test results for awt, swing and 2d.
>>
>>> Supposing once these tests were excluded from runs, there were some 
>>> causes for that e.g. some known product or test issues which were 
>>> not intended to be fixed for this particular release. My question is 
>>> how can I get to know these causes in order to differentiate 
>>> expected and unexpected failures - any of these tests, which were 
>>> excluded from EAB runs because of one issue, may fail because of 
>>> another.
>>>
>> With the exception of awt, swing and 2d, the test results or any 
>> differences from one build to another are highlighted in the  diff 
>> column.
>>
>>> I tried to follow the link at Summary column placed on the page 
>>> http://download.java.net/openjdk/testresults/9 and could not get 
>>> information about statuses of the tests (similar to OpenJDK8).
>>>
>> This link works for me, can you retry ?
>>>
>>> So another question, how can I get the list of regression tests (if 
>>> any) which expectedly fail for some particular OpenJDK release?
>>>
>> In the test results page - for 
>> examplehttp://download.java.net/openjdk/testresults/9/testresults.html
>>
>> The summary link provides a link to the tests that ran successfully. 
>> The diff column provides a link to the failures that are different to 
>> the previous build by component  - jdk, hotspot & langtools
>>
>> I hope that helps.
>>
>> Rgds,Rory
>>>
>>> Thanks,
>>>
>>> Vitaly
>>>
>>
>> -- 
>> Rgds,Rory O'Donnell
>> Quality Engineering Manager
>> Oracle EMEA, Dublin,Ireland
>

-- 
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20161004/12bc96dd/attachment-0001.html>