Regression test failures - could not find entries in JDK Bug System

[Vitaly Provodin]

Hi Rory,

Thanks a lot for the detailed answer - now it is clear for EAB runs. Sorry I overlooked the note 2.
Actually, the reason of my mail was to understand that the failures I got (including those two) are expected and can be explained by existing bugs. Totally there were about 300 test failures of java/awt and javax/swing tests in my environment. Most of them are explained by existing bugs.
For others (about 70) no bugs were found - most probably they are somehow mentioned in Bug System but i could find no entries.

Some of these 70 failures can be caused by specifics in my environment. But others like javax/swing/JPopupMenu/6800513/bug6800513.java may be considered as an issue in Java.
The main question here how can I report these issues (if I have no write-access to the Bug System)?
It looks like time-consuming job doing it via emails... 

What can you recommend me?

Thanks in advance,
Vitaly

P.S. now I am able to open number-links via http://download.java.net/openjdk/testresults/9/testresults.html <http://download.java.net/openjdk/testresults/9/testresults.html>
But they can not be opened if starting page is http://download.java.net/openjdk/testresults/9 <http://download.java.net/openjdk/testresults/9>

> On 04 Oct 2016, at 17:02, Rory O'Donnell <rory.odonnell at oracle.com> wrote:
> 
> Hi Vitaly,
> 
> A little bit of background, the purpose of the posting of EA build results is to provide consistent, reproducible
> results that people can generate and compare against in their own environment.
> 
> To this end, we do not run the awt, swing and 2d tests, as there are so many variables and the results 
> can be inconsistent.
> 
> On 04/10/2016 03:35, Vitaly Provodin wrote:
>> Hello,
>> 
>> My question relates to failures of the Regression tests (jdk/test) on OpenJDK8 and how to identify whether a failure is expected or not.
>> 
>> Running java/awt and javax/swing tests against OpenJDK8 on various platforms I got some failures. Trying to find any explanations for these failures I looked through the JDK Bug System and could not find any bugs explaining them, for example, for the following two tests
>> 
> As I mentioned above, we don't run these tests as part of the regular testing see NOTE 2 
> http://download.java.net/openjdk/testresults/9/testresults.html <http://download.java.net/openjdk/testresults/9/testresults.html>
> 2) ** Results do not includes test results for awt, swing and 2d.
>> 
>> -java/awt/Frame/UnfocusableMaximizedFrameResizablity/UnfocusableMaximizedFrameResizablity.java
>> 
>> Failed.
>> Program `/Users/builduser/buildAgent/work/13ab0d4c51b0de0b/jbsdk/jdk/Contents/Home/bin/java' timed out (timeout set to 240000ms, elapsed time was 240786ms).
>> 
>> - javax/swing/JPopupMenu/6800513/bug6800513.java
>> 
>> 
>> Failed.
>> Execution failed: `main' threw exception: java.lang.Exception: popup class is: javax.swing.PopupFactory$HeavyWeightPopup, expected: javax.swing.PopupFactory$LightWeightPopup
>> 
>> I found out that these tests were excluded from runs against JDK 8 Update Releases Early Access Build (EAB) - see Test Results page at http://download.java.net/openjdk/testresults/8 <http://download.java.net/openjdk/testresults/8>. Please note, I do not know the status of these results, whether they are actual for OpenJDK8 final release or not. Also, I paid attention to that most of the failures I am talking about are  not included into jdk/test/ProblemList.txt as for now.
>> 
> As I mentioned above, we don't run these tests as part of the regular testing see NOTE 2 
> http://download.java.net/openjdk/testresults/9/testresults.html <http://download.java.net/openjdk/testresults/9/testresults.html>
> 2) ** Results do not includes test results for awt, swing and 2d.
> 
>> Supposing once these tests were excluded from runs, there were some causes for that e.g. some known product or test issues which were not intended to be fixed for this particular release. My question is how can I get to know these causes in order to differentiate expected and unexpected failures - any of these tests, which were excluded from EAB runs because of one issue, may fail because of another.
>> 
> With the exception of awt, swing and 2d, the test results or any differences from one build to another are highlighted in the  diff column.
> 
>> I tried to follow the link at Summary column placed on the page http://download.java.net/openjdk/testresults/9 <http://download.java.net/openjdk/testresults/9> and could not get information about statuses of the tests (similar to OpenJDK8). 
>> 
> This link works for me, can you retry ?
>> 
>> So another question, how can I get the list of regression tests (if any) which expectedly fail for some particular OpenJDK release?
>> 
> In the test results page - for example http://download.java.net/openjdk/testresults/9/testresults.html <http://download.java.net/openjdk/testresults/9/testresults.html>
> 
> The summary link provides a link to the tests that ran successfully. The diff column provides a link to the failures that are different to the previous build by component  - jdk, hotspot & langtools
> 
> I hope that helps.
> 
> Rgds,Rory
>> Thanks,
>> 
>> Vitaly 
>> 
> 
> -- 
> Rgds,Rory O'Donnell
> Quality Engineering Manager
> Oracle EMEA, Dublin,Ireland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/quality-discuss/attachments/20161004/8f133165/attachment-0001.html>