Vector API testing infrastructure

Fri Feb 9 00:01:42 UTC 2018

Hi Jp,

Nice work, a very good start on something that is critical for this project.

I have some suggestions for incremental improvements then some more architectural comments to consider for further changes.

First, some incremental improvements:

- probably easier to assume that the project build tools, javac and java, are on the executable path otherwise you have to bind things to platform specific paths in the configuration.

- we can switch to depending on the testng jar supplied by jtreg [1], no other jar is required. However, because this version is quite old it is necessary to annotate the test classes with @Test in addition to the test methods for the command line execution to work.

So define a env variable say JTREG_PATH then TESTNG_JAR=$JTREG_PATH/lib/testng.jar, which can be used at compile and runtime.

- while we are at it lets add some jtreg meta-data to the classes:


+++ b/test/jdk/jdk/incubator/vector/testng/templates/header.template	Thu Feb 08 15:07:58 2018 -0800
@@ -23,6 +23,13 @@
  * questions.
  */
 
+/*
+ * @test
+ * @modules jdk.incubator.vector
+ * @run testng $vectorteststype$
+ *
+ */
+
 import jdk.incubator.vector.Shapes;
 import jdk.incubator.vector.Vector;
 
@@ -47,6 +54,7 @@
 
 import org.testng.Assert;
 
+ at org.testng.annotations.Test
 public class $vectorteststype$ {


then it’s possible to compile and run tests using jtreg e.g. from the root of the repository:

  jtreg -jdk:./build/macosx-x64/jdk/ -ea -esa -avm -va -nr  test/jdk/jdk/incubator/vector/testng/*VectorTests.java

then the test generation does not require compilation and you don’t require a separate run script, such functionality would be more of a convenience if you want to retain the ability to run without jtreg. However, in the long it will be jtreg that takes precedence as that is the means by which the tests will be run by the test infrastructure and developers.


Now, some architectural comments.

- I would like to see the use of TestNG data providers to supply the test data or functions that generate the test data. This makes it much easier to plugin in different data patterns, rather than hard code a single pattern into the test method itself.

- I think the template generation could be simplified if we further use data providers to create a cross product of data + operation rather than having a separate meta-template for each kind of operation.

Let’s consider the generation of an enum from a template for each operation kind.

The operations would be enumerated and have as properties functions for performing the scalar operation (accepting/return scalar arguments) and the vector operation (accepting accepting/return vector arguments).

A test method for an operation kind would be supplied arguments from an associated data provider, of these arguments one would be the enum value for the operation to be tested.

Here’s what it might concretely look like following the same test strategy:

@Test(dataProvider=“ Int512BinaryOpProvider”)
public void testBinaryOp(IntFunction<int[]> f1, IntFunction<int[]> f2, Int512BinaryOp op) {
    int[] a = f1.apply(SIZE);
    int[] b = f2.apply(SIZE);
    int[] c = new int[SIZE];

    // Computation.
    for (int i = 0; i < a.length; i += species.length()) {
        IntVector<Shapes.S512Bit> av = op.species().fromArray(a, i);
        IntVector<Shapes.S512Bit> bv = op.species().fromArray(b, i);
        op.apply(av, bv).intoArray(c, i);
    }

    // Checking.
    op.assertEquals(a, b, c);
}

enum Int512BinaryOp {
  ADD((a, b) -> a + b,
           (a, b) -> a.add(b));

  public interface ScalarBinaryOp {
      int apply(int i, int a, int b);
  }

  public interface VectorBinaryOp {
      IntVector<Shapes.S512Bit> apply(
          IntVector<Shapes.S512Bit> a, IntVector<Shapes.S512Bit> b);
  }

  Int512BinaryOp(ScalarBinaryOp sop, VectorBinaryOp tvop) { … }
  ...
}

It might even be possible make the test methods generic over the shape, alas generic enums are problematic, but the enums can implement a generic interface, and an instance of that interface is passed to the test method as an argument.

A downside of this approach is the actual intrinsic is hidden behind a function. I suspect this is not an issue with regards to actual generation of a vector hardware instruction but would like some confirmation, Razvan/Vladimir? 

If not it still might be an issue with regards to the effectiveness of vector box eliding and loop unrolling etc. But, i argue for such tests it does not matter as these are primarily functional tests not performance tests, which should be something separate that focus on using JMH.

If we need to generate specific operation tests i would still suggest using an enum for the scalar operations and test assertion, thus it can be shared across shapes. I like the generated code to be as if i would have written it by hand :-)

Further, i wonder if we could then use the enum values to drive test method generation, perhaps something to consider further on (pushing more of such logic into Java code rather than bash).

Paul.

[1] https://ci.adoptopenjdk.net/view/Dependencies/job/jtreg/


> On Feb 8, 2018, at 2:06 PM, Halimi, Jean-Philippe <jean-philippe.halimi at intel.com> wrote:
> 
> Thanks a lot for your feedback.
> 
> I agree with all of your points so far. I will work on that in the coming days.
> 
> Thanks to Razvan and Vivek's help, I published a new follow-up patch adding generated tests.
> 
> http://cr.openjdk.java.net/~rlupusoru/panama/webrev_vector_api_test_infra_01/
> 
> Cheers,
> 
> Jp
> 
> -----Original Message-----
> From: Vladimir Ivanov [mailto:vladimir.x.ivanov at oracle.com] 
> Sent: Thursday, February 8, 2018 1:38 PM
> To: Halimi, Jean-Philippe <jean-philippe.halimi at intel.com>; panama-dev at openjdk.java.net
> Cc: Viswanathan, Sandhya <sandhya.viswanathan at intel.com>
> Subject: Re: Vector API testing infrastructure
> 
> Nice work, Jp! Looks very good!
> 
>> It will be helpful to include the generated test cases as well.
> 
> I second that. Please, add generated test cases as well.
> 
> Some other comments:
> 
>   * testng in test path is useless
> 
> Do you see any problems with putting files right into test/jdk/jdk/incubator/vector/?
> 
>   * In the longer term, there should be only 1 script left: the one which generates test sources from templates.
> 
> Best regards,
> Vladimir Ivanov
> 
>> -----Original Message-----
>> From: panama-dev [mailto:panama-dev-bounces at openjdk.java.net] On 
>> Behalf Of Halimi, Jean-Philippe
>> Sent: Thursday, February 08, 2018 10:52 AM
>> To: panama-dev at openjdk.java.net
>> Subject: Vector API testing infrastructure
>> 
>> Hi,
>> 
>> I am a new contributor to this list. I am contributing Vector API testing framework. Please have a look at the webrev below to learn more about it. It features a README file in case you want to generate, build and run the tests.
>> The generated source code is not part of the initial commit, but they can be added if need be.
>> 
>> http://cr.openjdk.java.net/~rlupusoru/panama/webrev_vector_api_test_in
>> fra_00/
>> 
>> Cheers,
>> 
>> Jp
>> 

