Isolating tests from general concurrency

[Igor Ignatyev]

> On Mar 19, 2019, at 8:43 PM, Jonathan Gibbons <jonathan.gibbons at oracle.com> wrote:
> On 3/19/19 8:37 PM, Igor Ignatyev wrote:
>> 
>>> On Mar 19, 2019, at 7:52 PM, Jonathan Gibbons <jonathan.gibbons at oracle.com <mailto:jonathan.gibbons at oracle.com>> wrote:
>>> On 3/19/19 2:34 PM, Igor Ignatyev wrote:
>>>>> How is the queue managed when normally concurrent tests are launched?
>>>>> Instead of waiting til the end, a test requiring concurrency=1 could simply wait
>>>>> for the queue to drain before being launched.
>>>> I'll let Jon to comment on the current implementation. but waiting for the queue to drain each time we meet 'nonconcurrent' test will add multiple serialization points and might have measurable impact on overall execution time.
>>> Currently, jtreg uses the test runner mechanism provided by the underlying JT Harness framework.   Changing that will be a Very Big Deal.
>>> 
>>> How many nonconcurrent tests are there, out of how many tests total?
>>> 
>> if we count all current "exclusiveAccess" as nonconcurrent, we will get less than 4% of all tests, most of them are in hotspot test suites, and in hotspot test suites nonconcurrent tests make up 8%.
>> 
>> I prototyped smth similar to that Gary suggested in the dirtiest possible way[1] using read-write lock: nonconcurrent tests acquire a write lock, while other tests acquire read lock; will use it to measure performance impact for different test groups/directories later this week.
>> 
>> [1] http://cr.openjdk.java.net/~iignatyev//jtreg/rwLock/webrev.00/index.html <http://cr.openjdk.java.net/~iignatyev//jtreg/rwLock/webrev.00/index.html> 
>>> 
>>> -- Jon
>>> 
>> 
> Sounds good.
> 
> I'll also talk to Dmitry Bessonov about what's available in JT Harness.
> 
> -- Jon
> 

as promised, I have run some test groups w/ and w/o [1] (above) to compare the numbers. I used the host w/ Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz and used 'make test' to run jtreg. make adds "-conc:31" on this host, so the impact is very pronounced (and can be even nonrepresentational):

#<test group> : <new time / old time>  
vmTestbase_nsk_jvmti : 3.26
vmTestbase_nsk_jdi : 2.61
jdk/tier3 : 1.64
vmTestbase_vm_gc : 1.45
jdk/tier1 : 1.40
jdk_svc : 1.17
vmTestbase_nsk_monitoring : 1.09
jdk/tier2 : 1.05
vmTestbase_vm_metaspace : 1.02
vmTestbase_nsk_stress : 1.01
vmTestbase_vm_mlvm : 1.00
vmTestbase_nsk_sysdict : 1.00

as you can see, the most impacted test groups are jvmti and jdi, jvmti has only 22 noncurrent tests out of 651, and jdi has 28 of 1143. the least impact groups -- sysdict and mlvm have 20 of 20 and 46 of 63 respectively.

Thanks,
-- Igor