VectorAPI: Test plan draft

[Paul Sandoz]

Looks good, a small typo below.

> On Jul 3, 2018, at 8:09 AM, Halimi, Jean-Philippe <jean-philippe.halimi at intel.com> wrote:
> 
> Dear all,
> 
> Here is the updated version. Let me know if there's anything else you'd like to change.
> 
> Introduction
> -	There are 6 types and 4 different vector sizes supported:
> 	o	Types: byte (8 bits), short (16 bits), int (32 bits), long (64 bits), float (32 bits), double (64 bits)
> 	o	Vector sizes: 64 bits, 128 bits, 256 bits, 512 bits
> -	x86 intrinsification for all tests realized in AVX and in some cases SSE
> 	o	Generation of hardware-accelerated vector code sequences on supported hardware
> 	o	Dynamic code dispatch depending on underlying ISA
> 
> Test success criteria
> 1.	All intrinsified VectorAPI methods are covered
> 2.	All vector shapes and sizes combinations are covered
> 3.	Test pass rate is 100% when tested on Skylake Server with default configuration and with HotSpot options covering various subsets of supported instructions
> 4.	Test success stability is 99%
> 
> Test methodology
> The test infrastructure consists of files and scripts which goal is to prepare and run the tests, and visualize the test results.

s/which goal is to/that/

Paul.

> -	Template files factorize tests by type (Unary, Binary etc)
> -	Shell script generates test bits for each vector shape and size
> 	o	gen-template.sh details all the supported tests.
> 	o	gen-tests.sh is used to generate all the tests files.
> -	The test infrastructure uses jtreg to launch the tests
> -	Each test:
> 	o	Has a reference Java implementation of the Vector API feature being tested.
> 	o	Takes functors as inputs used to mimics a set of realistic use-case scenario
> 	o	Asserts that the result of the Vector API call matches the reference implementation
> -	In case an assertion is failing, the test will generate an exception
> 	o	Details about incorrect calculations or crashes will be available for later use
> 
> Test Inventory
> Two Java source files are generated per vector shape automatically by a template mechanism.
> Example: Integers vectors, 128bits long:
> -	Int128VectorTests.java: contains all API calls-related tests
> -	Int128VectorLoadStoreTests.java: contains all Load/Store-related tests
> 
> Test configurations
> Testing of one specific configuration has a reasonable time (<2 minutes). However, several configurations should be covered, including:
> -	Testing on Linux, Windows and MacOS
> -	Testing with -XX:+UseVectorApiIntrinsics and also with -XX:-UseVectorApiIntrinsics
> 	o	The latter will ensure the Java implementations are correct
> -	Testing with every combination of UseSSE=0 through UseSSE=4 and UseAVX=0 through UseAVX=3
> 	o	Only test these options with -XX:+UseVectorApiIntrinsics enabled
> 
> Jp
> 
> -----Original Message-----
> From: Paul Sandoz [mailto:paul.sandoz at oracle.com] 
> Sent: Thursday, June 28, 2018 11:47 AM
> To: Halimi, Jean-Philippe <jean-philippe.halimi at intel.com>
> Cc: panama-dev at openjdk.java.net
> Subject: Re: VectorAPI: Test plan draft
> 
> Hi Jp,
> 
> Thanks for writing this up. Overall it looks good.
> 
>> On Jun 22, 2018, at 9:20 AM, Halimi, Jean-Philippe <jean-philippe.halimi at intel.com> wrote:
>> 
>> Dear all,
>> 
>> Here is the first draft of the Vector API test plan. Can you please review it and let me know your thoughts?
>> 
>> Thank you,
>> 
>> Jp
>> 
>> --
>> 
>> Introduction
>> 
>> -          There are 6 types and 4 different vector sizes supported:
>> 
>> o   Types: byte (8 bits), short (16 bits), int (32 bits), long (64 bits), float (32 bits), double (64 bits)
>> 
>> o   Vector sizes: 64 bits, 128 bits, 256 bits, 512 bits
>> 
>> -          x86 intrinsification for all tests realized in AVX and in some cases SSE
>> 
>> o   Generation of hardware-accelerated vector code sequences on supported hardware
>> 
>> o   Dynamic code dispatch depending on underlying ISA
>> Test success criteria
>> 
>> 1.       All intrinsified VectorAPI methods are covered
>> 
>> 2.       All vector shapes and sizes combinations are covered
>> 
>> 3.       Test pass rate is 100% when tested on Skylake Server with default configuration
> 
> Ok, with the HotSpot options that should cover the various subsets of supported instructions on other kinds of server.
> 
> 
>> 
>> 4.       Test success stability is 99%
> 
>> Test methodology
>> The test infrastructure consists of files and scripts which goal is to prepare and run the tests, and visualize the test results.
>> 
>> -          Template files factorize tests by type (Unary, Binary etc)
>> 
>> -          Shell script generates test bits for each vector shape and size
>> 
>> o   gen-template.sh details all the supported tests.
>> 
>> o   gen-tests.sh is used to generate all the tests files.
>> 
>> -          The test infrastructure uses jtreg to launch the tests
>> 
>> -          Each test:
>> 
>> o   Has a reference Java implementation of the Vector API feature being tested.
>> 
>> o   Takes functors as inputs used to mimics a set of realistic use-case scenario
>> 
>> o   Asserts that the result of the Vector API call matches the reference implementation
>> 
>> -          In case an assertion is failing, the test will generate an exception
>> 
>> o   Details about incorrect calculations or crashes will be available for later use
>> Test Inventory
>> Two Java source files are generated per vector shape automatically by a template mechanism.
>> Example: Integers vectors, 128bits long:
>> 
>> -          Int128VectorTests.java: contains all API calls-related tests
>> 
>> -          Int128VectorLoadStoreTests.java: contains all Load/Store-related tests
>> Test configurations
>> Testing should cover:
>> 
>> -          Testing on both Linux and Windows
> 
> Mac too?
> 
> 
>> 
>> -          Testing with every combination of UseSSE=0 through UseSSE=4 and UseAVX=0 through UseAVX=3
> 
> Only need to test those combinations with -XX:+UseVectorApiIntrinsics?
> 
> 
>> 
>> -          Testing with -XX:+UseVectorApiIntrinsics and also with -XX:-UseVectorApiIntrinsics
> 
> The latter will ensure testing of the Java implementations.
> 
> 
> It would be useful to mention that test execution time for one whole run under a single set of arguments is reasonable (e.g. < 2 minutes say) with possible improvements.
> 
> Thanks,
> Paul.