RFR 8184445: JShell tests: fail intermittently if tests are run in high concurrent mode.

[Joseph D. Darcy]

Hi Robert,

I'd prefer if only the launching tests and other known failures were 
segregated into a non-concurrent area. That would still let ~3/4 of the 
tests proceed normally.

As a follow-up, can an RFE be filed to harden the intermittently failing 
tests against concurrent networking?

Thanks,

-Joe

On 2/27/2018 12:21 PM, Robert Field wrote:
> OK, I did a survey of all the JShell bugs.  There are over a dozen 
> intermittent test failures, almost all are probably network related.  
> But if we limit to just intermittent failures to launch, then there 
> are seven.
>
> There are 17 tests of launching configuration, and 75 'normal' tests.  
> So, the launching configuration tests do fail disproportionately, 3 
> mentioned failures vs 5 mentioned failing files.
>
> The bug that highlighted the concurrent testing -- "JShell tests: fail 
> intermittently if tests are run in high concurrent mode":
>     https://bugs.openjdk.java.net/browse/JDK-8184445
> mentioned 'several' issues, the two included JTR files are, tellingly, 
> normal tests.
>
> The non-launching intermittent failures are all normal tests.
>
> So, where does that leave us?  I could reduce the failures a bit at 
> low time-cost by putting the launching configuration tests in the 
> exclusiveAccess.dirs.  Or, I could, at considerable testing cost, 
> address the broad swath.
>
> -Robert
>
> On 02/26/18 17:28, joe darcy wrote:
>> Hi Robert,
>>
>> On 2/26/2018 10:57 AM, Robert Field wrote:
>>>
>>>
>>> On 02/26/18 10:23, joe darcy wrote:
>>>> Hi Robert,
>>>>
>>>> The fix looks acceptable in terms of addressing the problem, but is 
>>>> there a sense of how this might impact running time of the test suite?
>>>>
>>>> Phrased differently, are there plans to make the tests more robust 
>>>> to concurrent runs in the future?
>>>
>>> Hi Joe,
>>>
>>> There is a lot of network connection happening in these tests, most 
>>> of which is in layers we don't control (JDI).  We have been trying 
>>> to lower the risk and we don't see failures running the tests 
>>> ourselves, but intermittent failures scattered through the suite 
>>> during testing (e.g. mach5) have been a constant problem.
>>>
>>> We will see the impact on test duration.  Default connection has 
>>> three-level fail-over, the tests of other connection modes see 
>>> failure far more frequently, so, if necessary, we can look at tuning 
>>> this.
>>>
>>
>> From some quick checking, there are about 80 tests in that directory. 
>> From one sample point on my laptop, the tests took a good long while 
>> to run. If some of the tests can be reliably run concurrently, I'd 
>> much prefer to see a subset of tests moved to a sheltered directory.
>>
>> Thanks,
>>
>> -Joe
>