VectorAPI: Test plan draft

[Halimi, Jean-Philippe]

Dear all,

Here is the first draft of the Vector API test plan. Can you please review it and let me know your thoughts?

Thank you,

Jp

--

Introduction

-          There are 6 types and 4 different vector sizes supported:

o   Types: byte (8 bits), short (16 bits), int (32 bits), long (64 bits), float (32 bits), double (64 bits)

o   Vector sizes: 64 bits, 128 bits, 256 bits, 512 bits

-          x86 intrinsification for all tests realized in AVX and in some cases SSE

o   Generation of hardware-accelerated vector code sequences on supported hardware

o   Dynamic code dispatch depending on underlying ISA
Test success criteria

1.       All intrinsified VectorAPI methods are covered

2.       All vector shapes and sizes combinations are covered

3.       Test pass rate is 100% when tested on Skylake Server with default configuration

4.       Test success stability is 99%
Test methodology
The test infrastructure consists of files and scripts which goal is to prepare and run the tests, and visualize the test results.

-          Template files factorize tests by type (Unary, Binary etc)

-          Shell script generates test bits for each vector shape and size

o   gen-template.sh details all the supported tests.

o   gen-tests.sh is used to generate all the tests files.

-          The test infrastructure uses jtreg to launch the tests

-          Each test:

o   Has a reference Java implementation of the Vector API feature being tested.

o   Takes functors as inputs used to mimics a set of realistic use-case scenario

o   Asserts that the result of the Vector API call matches the reference implementation

-          In case an assertion is failing, the test will generate an exception

o   Details about incorrect calculations or crashes will be available for later use
Test Inventory
Two Java source files are generated per vector shape automatically by a template mechanism.
Example: Integers vectors, 128bits long:

-          Int128VectorTests.java: contains all API calls-related tests

-          Int128VectorLoadStoreTests.java: contains all Load/Store-related tests
Test configurations
Testing should cover:

-          Testing on both Linux and Windows

-          Testing with every combination of UseSSE=0 through UseSSE=4 and UseAVX=0 through UseAVX=3

-          Testing with -XX:+UseVectorApiIntrinsics and also with -XX:-UseVectorApiIntrinsics