Vector API testing infrastructure

Tue Feb 13 17:06:51 UTC 2018

> On Feb 12, 2018, at 1:34 PM, Lupusoru, Razvan A <razvan.a.lupusoru at intel.com> wrote:
> 
> Hi JP,
> 
> My thoughts so far:
> - I would like that we keep existing .sh scripts even when though running through jtreg is possible. For debugging, I have found that fewer launching layers simplify being able to break at and debug generated code. Thus having the scripts in place will allow me to make sure I have all the right command line options to properly exercise the features.

See my previous email to Jp on jtreg and it’s verbose option, it may get you some of the way,  but keeping the run script around for now is entirely reasonable.

> - I would be happy to get the tests merged as soon as possible with your latest updates so that infrastructure is up - thus future commits of intrinsification features should also be making additions as needed to test infrastructure in order to get the coverage.

Agreed.

> - The ideas from Paul will likely be a positive for maintenance of the tests. My only concern is that it adds additional abstraction and thus requires more reliance that Hotspot will do expected thing and inline and not box unnecessarily. I personally like the current templating mechanism because the generated tests end up showing very specifically the API methods being exercised and are useful for analyzing and testing generated code.

Others have also made similar points, let’s keep it more explicit and less abstract.

Paul.

> -Enhancement of infrastructure to use data providers is a good idea so that we can have multiple generators for inputs which will allow testing of normal inputs and also edge case inputs.
> 
> Thanks,
> Razvan
> 
> -----Original Message-----
> From: panama-dev [mailto:panama-dev-bounces at openjdk.java.net] On Behalf Of Halimi, Jean-Philippe
> Sent: Monday, February 12, 2018 1:14 PM
> To: Paul Sandoz <paul.sandoz at oracle.com>; Vladimir Ivanov <vladimir.x.ivanov at oracle.com>
> Cc: panama-dev at openjdk.java.net
> Subject: RE: Vector API testing infrastructure
> 
> Hi all,
> 
> 
> 
> Thanks a lot for your feedback. It's great to see the enthusiasm around this project! ☺ I have already addressed some of your concerns regarding the Testing API. I’m trying here to summarize where we are and what is next.
> 
> 
> 
> Right now, the situation is that we have:
> 
> ·         A bunch of features we want to add
> 
> ·         A bunch of design modifications we want to perform
> 
> ·         A bunch of tests to be added/written by various people
> 
> 
> 
> Here is a summary of the feedback I have so far. Feel free to complete/edit if I missed anything. ☺
> 
> 1.       Jtreg integration
> 
> 2.       Java/javac environment-defined
> 
> 3.       Abstract data generation in a user-defined into a data provider
> 
> 4.       Command-line controlled vector size
> 
> 
> 
> I have already taken care of the 2 first items and it seems to work as is with jtreg. Here are now my questions:
> 
> 1.       Do you think it is reasonable to merge the code as is (as soon as I get my current webrev ready), so that people can get familiar with the interface, and then proceed with the rest?
> 
> 2.       Now that we can run the tests with jtreg, do you think we should keep the run_tests.sh script or just let it to the user to use his favorite jtreg command?
> 
> 
> 
> Please let me know your thoughts. ☺
> 
> 
> 
> Thanks,
> 
> 
> 
> Jp
> 
> 
> 
> -----Original Message-----
> From: Paul Sandoz [mailto:paul.sandoz at oracle.com]
> Sent: Friday, February 9, 2018 9:35 AM
> To: Vladimir Ivanov <vladimir.x.ivanov at oracle.com>
> Cc: Halimi, Jean-Philippe <jean-philippe.halimi at intel.com>; panama-dev at openjdk.java.net
> Subject: Re: Vector API testing infrastructure
> 
> 
> 
> 
> 
>> On Feb 9, 2018, at 3:27 AM, Vladimir Ivanov <vladimir.x.ivanov at oracle.com<mailto:vladimir.x.ivanov at oracle.com>> wrote:
> 
>> 
> 
>> 
> 
>>> It might even be possible make the test methods generic over the shape, alas generic enums are problematic, but the enums can implement a generic interface, and an instance of that interface is passed to the test method as an argument.
> 
>> 
> 
>> You can push it even further and end up without any template generation at all, only with abstract code shapes. For example, operations can be represented as MethodHandles and applied using MH.invoke(), arrays erased to Objects.
> 
>> 
> 
>>> A downside of this approach is the actual intrinsic is hidden behind a function. I suspect this is not an issue with regards to actual generation of a vector hardware instruction but would like some confirmation, Razvan/Vladimir?
> 
>> 
> 
>> Intrinsification will reliably happen with generic intrinsics, but for original implementation it depends on what is passed in as Int512BinaryOp. If it's a lambda which refers to concrete types, then intrinsification will happen as well. But no inlining in both cases.
> 
>> 
> 
> 
> 
> Ok, that is what i figured.
> 
> 
> 
> 
> 
>>> If not it still might be an issue with regards to the effectiveness of vector box eliding and loop unrolling etc. But, i argue for such tests it does not matter as these are primarily functional tests not performance tests, which should be something separate that focus on using JMH.
> 
>> 
> 
>> If Vector API were a JDK-only feature, I'd agree with you here. But we have substantial portion of code in JVM and we need to cover it with tests as well. Benchmarks have completely different purpose than unit tests and having 2 (mostly duplicating) sets of tests for JDK & JVM parts looks like an overkill.
> 
>> 
> 
> 
> 
> I suspect some form of duplication between unit and JMH tests will be unavoidable, but i don’t think the later needs to be as exhaustive, the generative technique can be similar for both though that then it might be easy to generate a JMH test on demand for a type, shape and operation.
> 
> 
> 
> 
> 
>> So, I'd prefer to see the tests to be both JDK- & JVM-friendly: checking correctness with code shapes amenable to optimizations.
> 
>> 
> 
> 
> 
> Point accepted on more literal being beneficial to the JVM.
> 
> 
> 
> There is also an argument not to lean all the way on data providers and splitting out functionality into separate classes since it is easier to perform selective execution and each test runs in a smaller amount of time.
> 
> 
> 
> I’ll noodle more on the operation-kind method generation but this architectural discussion should not block integration of the tests (after addressing my proposed incremental improvements).
> 
> 
> 
> Thanks,
> 
> Paul.
> 
> 