Vector API testing infrastructure

[Halimi, Jean-Philippe]

Hi all,



Thanks a lot for your feedback. It's great to see the enthusiasm around this project! ☺ I have already addressed some of your concerns regarding the Testing API. I’m trying here to summarize where we are and what is next.



Right now, the situation is that we have:

·         A bunch of features we want to add

·         A bunch of design modifications we want to perform

·         A bunch of tests to be added/written by various people



Here is a summary of the feedback I have so far. Feel free to complete/edit if I missed anything. ☺

1.       Jtreg integration

2.       Java/javac environment-defined

3.       Abstract data generation in a user-defined into a data provider

4.       Command-line controlled vector size



I have already taken care of the 2 first items and it seems to work as is with jtreg. Here are now my questions:

1.       Do you think it is reasonable to merge the code as is (as soon as I get my current webrev ready), so that people can get familiar with the interface, and then proceed with the rest?

2.       Now that we can run the tests with jtreg, do you think we should keep the run_tests.sh script or just let it to the user to use his favorite jtreg command?



Please let me know your thoughts. ☺



Thanks,



Jp



-----Original Message-----
From: Paul Sandoz [mailto:paul.sandoz at oracle.com]
Sent: Friday, February 9, 2018 9:35 AM
To: Vladimir Ivanov <vladimir.x.ivanov at oracle.com>
Cc: Halimi, Jean-Philippe <jean-philippe.halimi at intel.com>; panama-dev at openjdk.java.net
Subject: Re: Vector API testing infrastructure





> On Feb 9, 2018, at 3:27 AM, Vladimir Ivanov <vladimir.x.ivanov at oracle.com<mailto:vladimir.x.ivanov at oracle.com>> wrote:

>

>

>> It might even be possible make the test methods generic over the shape, alas generic enums are problematic, but the enums can implement a generic interface, and an instance of that interface is passed to the test method as an argument.

>

> You can push it even further and end up without any template generation at all, only with abstract code shapes. For example, operations can be represented as MethodHandles and applied using MH.invoke(), arrays erased to Objects.

>

>> A downside of this approach is the actual intrinsic is hidden behind a function. I suspect this is not an issue with regards to actual generation of a vector hardware instruction but would like some confirmation, Razvan/Vladimir?

>

> Intrinsification will reliably happen with generic intrinsics, but for original implementation it depends on what is passed in as Int512BinaryOp. If it's a lambda which refers to concrete types, then intrinsification will happen as well. But no inlining in both cases.

>



Ok, that is what i figured.





>> If not it still might be an issue with regards to the effectiveness of vector box eliding and loop unrolling etc. But, i argue for such tests it does not matter as these are primarily functional tests not performance tests, which should be something separate that focus on using JMH.

>

> If Vector API were a JDK-only feature, I'd agree with you here. But we have substantial portion of code in JVM and we need to cover it with tests as well. Benchmarks have completely different purpose than unit tests and having 2 (mostly duplicating) sets of tests for JDK & JVM parts looks like an overkill.

>



I suspect some form of duplication between unit and JMH tests will be unavoidable, but i don’t think the later needs to be as exhaustive, the generative technique can be similar for both though that then it might be easy to generate a JMH test on demand for a type, shape and operation.





> So, I'd prefer to see the tests to be both JDK- & JVM-friendly: checking correctness with code shapes amenable to optimizations.

>



Point accepted on more literal being beneficial to the JVM.



There is also an argument not to lean all the way on data providers and splitting out functionality into separate classes since it is easier to perform selective execution and each test runs in a smaller amount of time.



I’ll noodle more on the operation-kind method generation but this architectural discussion should not block integration of the tests (after addressing my proposed incremental improvements).



Thanks,

Paul.