Isolating tests from general concurrency

[Stefan Karlsson]

Hi all,

I'd like to ping this thread. We have some very hard-to-debug timeouts 
on windows, and getting something like this in place will help rule out 
some potential problems.

Thanks,
StefanK

On 2021-02-25 00:00, Igor Ignatyev wrote:
> Hi Jon and all,
>
> I'd like to resurrect this thread as this still bites us.
>
> I've filed an RFE[1], rebased my prototype and put it on github[2]. as 
> before, it changes `exclusiveAccess` to mean  "won't be run 
> concurrently with any other tests" (as opposed to the current "won't 
> be run concurrently with other `exclusiveAccess ` tests"), from my 
> point of view, it is better than introducing of a new k/w as it's less 
> confusing and it doesn't seem to be that beneficial (as in many cases 
> people want their `exclusiveAccess` tests to have a real exclusiveness).
>
> unless JT Harness provides us with a better way to achieve a full 
> exclusiveness, I'll go ahead, polish my patch and send an official 
> RFR. that's, obviously, if we agree that it's a way to go (at least 
> for now).
>
> [1] https://bugs.openjdk.java.net/browse/CODETOOLS-7902831 
> <https://bugs.openjdk.java.net/browse/CODETOOLS-7902831>
> [2] https://github.com/openjdk/jtreg/pull/4 
> <https://github.com/openjdk/jtreg/pull/4>
>
> Thanks,
> -- Igor
>
>> On Mar 25, 2019, at 12:00 PM, Igor Ignatyev <igor.ignatyev at oracle.com 
>> <mailto:igor.ignatyev at oracle.com>> wrote:
>>
>>
>>> On Mar 19, 2019, at 8:43 PM, Jonathan Gibbons 
>>> <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>> 
>>> wrote:
>>> On 3/19/19 8:37 PM, Igor Ignatyev wrote:
>>>>
>>>>> On Mar 19, 2019, at 7:52 PM, Jonathan Gibbons 
>>>>> <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>> 
>>>>> wrote:
>>>>> On 3/19/19 2:34 PM, Igor Ignatyev wrote:
>>>>>>> How is the queue managed when normally concurrent tests are launched?
>>>>>>> Instead of waiting til the end, a test requiring concurrency=1 could simply wait
>>>>>>> for the queue to drain before being launched.
>>>>>> I'll let Jon to comment on the current implementation. but waiting for the queue to drain each time we meet 'nonconcurrent' test will add multiple serialization points and might have measurable impact on overall execution time.
>>>>>
>>>>> Currently, jtreg uses the test runner mechanism provided by the 
>>>>> underlying JT Harness framework.   Changing that will be a Very 
>>>>> Big Deal.
>>>>>
>>>>> How many nonconcurrent tests are there, out of how many tests total?
>>>>>
>>>> if we count all current "exclusiveAccess" as nonconcurrent, we will 
>>>> get less than 4% of all tests, most of them are in hotspot test 
>>>> suites, and in hotspot test suites nonconcurrent tests make up 8%.
>>>>
>>>> I prototyped smth similar to that Gary suggested in the dirtiest 
>>>> possible way[1] using read-write lock: nonconcurrent tests acquire 
>>>> a write lock, while other tests acquire read lock; will use it to 
>>>> measure performance impact for different test groups/directories 
>>>> later this week.
>>>>
>>>> [1] 
>>>> http://cr.openjdk.java.net/~iignatyev//jtreg/rwLock/webrev.00/index.html 
>>>> <http://cr.openjdk.java.net/~iignatyev//jtreg/rwLock/webrev.00/index.html> 
>>>>
>>>>>
>>>>> -- Jon
>>>>>
>>>>
>>> Sounds good.
>>>
>>> I'll also talk to Dmitry Bessonov about what's available in JT Harness.
>>>
>>> -- Jon
>>>
>>
>> as promised, I have run some test groups w/ and w/o [1] (above) to 
>> compare the numbers. I used the host w/ Intel(R) Xeon(R) Platinum 
>> 8168 CPU @ 2.70GHz and used 'make test' to run jtreg. make adds 
>> "-conc:31" on this host, so the impact is very pronounced (and can be 
>> even nonrepresentational):
>>
>> #<test group> : <new time / old time>
>> vmTestbase_nsk_jvmti : 3.26
>> vmTestbase_nsk_jdi : 2.61
>> jdk/tier3 : 1.64
>> vmTestbase_vm_gc : 1.45
>> jdk/tier1 : 1.40
>> jdk_svc : 1.17
>> vmTestbase_nsk_monitoring : 1.09
>> jdk/tier2 : 1.05
>> vmTestbase_vm_metaspace : 1.02
>> vmTestbase_nsk_stress : 1.01
>> vmTestbase_vm_mlvm : 1.00
>> vmTestbase_nsk_sysdict : 1.00
>>
>> as you can see, the most impacted test groups are jvmti and jdi, 
>> jvmti has only 22 noncurrent tests out of 651, and jdi has 28 of 
>> 1143. the least impact groups -- sysdict and mlvm have 20 of 20 and 
>> 46 of 63 respectively.
>>
>> Thanks,
>> -- Igor
>>
>