VectorAPI: Test plan draft

[Paul Sandoz]

Hi Jp,

Thanks for writing this up. Overall it looks good.

> On Jun 22, 2018, at 9:20 AM, Halimi, Jean-Philippe <jean-philippe.halimi at intel.com> wrote:
> 
> Dear all,
> 
> Here is the first draft of the Vector API test plan. Can you please review it and let me know your thoughts?
> 
> Thank you,
> 
> Jp
> 
> --
> 
> Introduction
> 
> -          There are 6 types and 4 different vector sizes supported:
> 
> o   Types: byte (8 bits), short (16 bits), int (32 bits), long (64 bits), float (32 bits), double (64 bits)
> 
> o   Vector sizes: 64 bits, 128 bits, 256 bits, 512 bits
> 
> -          x86 intrinsification for all tests realized in AVX and in some cases SSE
> 
> o   Generation of hardware-accelerated vector code sequences on supported hardware
> 
> o   Dynamic code dispatch depending on underlying ISA
> Test success criteria
> 
> 1.       All intrinsified VectorAPI methods are covered
> 
> 2.       All vector shapes and sizes combinations are covered
> 
> 3.       Test pass rate is 100% when tested on Skylake Server with default configuration

Ok, with the HotSpot options that should cover the various subsets of supported instructions on other kinds of server.


> 
> 4.       Test success stability is 99%

> Test methodology
> The test infrastructure consists of files and scripts which goal is to prepare and run the tests, and visualize the test results.
> 
> -          Template files factorize tests by type (Unary, Binary etc)
> 
> -          Shell script generates test bits for each vector shape and size
> 
> o   gen-template.sh details all the supported tests.
> 
> o   gen-tests.sh is used to generate all the tests files.
> 
> -          The test infrastructure uses jtreg to launch the tests
> 
> -          Each test:
> 
> o   Has a reference Java implementation of the Vector API feature being tested.
> 
> o   Takes functors as inputs used to mimics a set of realistic use-case scenario
> 
> o   Asserts that the result of the Vector API call matches the reference implementation
> 
> -          In case an assertion is failing, the test will generate an exception
> 
> o   Details about incorrect calculations or crashes will be available for later use
> Test Inventory
> Two Java source files are generated per vector shape automatically by a template mechanism.
> Example: Integers vectors, 128bits long:
> 
> -          Int128VectorTests.java: contains all API calls-related tests
> 
> -          Int128VectorLoadStoreTests.java: contains all Load/Store-related tests
> Test configurations
> Testing should cover:
> 
> -          Testing on both Linux and Windows

Mac too?


> 
> -          Testing with every combination of UseSSE=0 through UseSSE=4 and UseAVX=0 through UseAVX=3

Only need to test those combinations with -XX:+UseVectorApiIntrinsics?


> 
> -          Testing with -XX:+UseVectorApiIntrinsics and also with -XX:-UseVectorApiIntrinsics

The latter will ensure testing of the Java implementations.


It would be useful to mention that test execution time for one whole run under a single set of arguments is reasonable (e.g. < 2 minutes say) with possible improvements.

Thanks,
Paul.