RFR 8184445: JShell tests: fail intermittently if tests are run in high concurrent mode.

[Robert Field]

OK, I did a survey of all the JShell bugs.  There are over a dozen 
intermittent test failures, almost all are probably network related.  
But if we limit to just intermittent failures to launch, then there are 
seven.

There are 17 tests of launching configuration, and 75 'normal' tests.  
So, the launching configuration tests do fail disproportionately, 3 
mentioned failures vs 5 mentioned failing files.

The bug that highlighted the concurrent testing -- "JShell tests: fail 
intermittently if tests are run in high concurrent mode":
     https://bugs.openjdk.java.net/browse/JDK-8184445
mentioned 'several' issues, the two included JTR files are, tellingly, 
normal tests.

The non-launching intermittent failures are all normal tests.

So, where does that leave us?  I could reduce the failures a bit at low 
time-cost by putting the launching configuration tests in the 
exclusiveAccess.dirs.  Or, I could, at considerable testing cost, 
address the broad swath.

-Robert

On 02/26/18 17:28, joe darcy wrote:
> Hi Robert,
>
> On 2/26/2018 10:57 AM, Robert Field wrote:
>>
>>
>> On 02/26/18 10:23, joe darcy wrote:
>>> Hi Robert,
>>>
>>> The fix looks acceptable in terms of addressing the problem, but is 
>>> there a sense of how this might impact running time of the test suite?
>>>
>>> Phrased differently, are there plans to make the tests more robust 
>>> to concurrent runs in the future?
>>
>> Hi Joe,
>>
>> There is a lot of network connection happening in these tests, most 
>> of which is in layers we don't control (JDI).  We have been trying to 
>> lower the risk and we don't see failures running the tests ourselves, 
>> but intermittent failures scattered through the suite during testing 
>> (e.g. mach5) have been a constant problem.
>>
>> We will see the impact on test duration.  Default connection has 
>> three-level fail-over, the tests of other connection modes see 
>> failure far more frequently, so, if necessary, we can look at tuning 
>> this.
>>
>
> From some quick checking, there are about 80 tests in that directory. 
> From one sample point on my laptop, the tests took a good long while 
> to run. If some of the tests can be reliably run concurrently, I'd 
> much prefer to see a subset of tests moved to a sheltered directory.
>
> Thanks,
>
> -Joe