VectorAPI: Test plan draft

[Halimi, Jean-Philippe]

Dear all,

Here is the updated version. Let me know if there's anything else you'd like to change.

Introduction
-	There are 6 types and 4 different vector sizes supported:
	o	Types: byte (8 bits), short (16 bits), int (32 bits), long (64 bits), float (32 bits), double (64 bits)
	o	Vector sizes: 64 bits, 128 bits, 256 bits, 512 bits
-	x86 intrinsification for all tests realized in AVX and in some cases SSE
	o	Generation of hardware-accelerated vector code sequences on supported hardware
	o	Dynamic code dispatch depending on underlying ISA

Test success criteria
1.	All intrinsified VectorAPI methods are covered
2.	All vector shapes and sizes combinations are covered
3.	Test pass rate is 100% when tested on Skylake Server with default configuration and with HotSpot options covering various subsets of supported instructions
4.	Test success stability is 99%

Test methodology
The test infrastructure consists of files and scripts which goal is to prepare and run the tests, and visualize the test results.
-	Template files factorize tests by type (Unary, Binary etc)
-	Shell script generates test bits for each vector shape and size
	o	gen-template.sh details all the supported tests.
	o	gen-tests.sh is used to generate all the tests files.
-	The test infrastructure uses jtreg to launch the tests
-	Each test:
	o	Has a reference Java implementation of the Vector API feature being tested.
	o	Takes functors as inputs used to mimics a set of realistic use-case scenario
	o	Asserts that the result of the Vector API call matches the reference implementation
-	In case an assertion is failing, the test will generate an exception
	o	Details about incorrect calculations or crashes will be available for later use

Test Inventory
Two Java source files are generated per vector shape automatically by a template mechanism.
Example: Integers vectors, 128bits long:
-	Int128VectorTests.java: contains all API calls-related tests
-	Int128VectorLoadStoreTests.java: contains all Load/Store-related tests

Test configurations
Testing of one specific configuration has a reasonable time (<2 minutes). However, several configurations should be covered, including:
-	Testing on Linux, Windows and MacOS
-	Testing with -XX:+UseVectorApiIntrinsics and also with -XX:-UseVectorApiIntrinsics
	o	The latter will ensure the Java implementations are correct
-	Testing with every combination of UseSSE=0 through UseSSE=4 and UseAVX=0 through UseAVX=3
	o	Only test these options with -XX:+UseVectorApiIntrinsics enabled

Jp

-----Original Message-----
From: Paul Sandoz [mailto:paul.sandoz at oracle.com] 
Sent: Thursday, June 28, 2018 11:47 AM
To: Halimi, Jean-Philippe <jean-philippe.halimi at intel.com>
Cc: panama-dev at openjdk.java.net
Subject: Re: VectorAPI: Test plan draft

Hi Jp,

Thanks for writing this up. Overall it looks good.

> On Jun 22, 2018, at 9:20 AM, Halimi, Jean-Philippe <jean-philippe.halimi at intel.com> wrote:
> 
> Dear all,
> 
> Here is the first draft of the Vector API test plan. Can you please review it and let me know your thoughts?
> 
> Thank you,
> 
> Jp
> 
> --
> 
> Introduction
> 
> -          There are 6 types and 4 different vector sizes supported:
> 
> o   Types: byte (8 bits), short (16 bits), int (32 bits), long (64 bits), float (32 bits), double (64 bits)
> 
> o   Vector sizes: 64 bits, 128 bits, 256 bits, 512 bits
> 
> -          x86 intrinsification for all tests realized in AVX and in some cases SSE
> 
> o   Generation of hardware-accelerated vector code sequences on supported hardware
> 
> o   Dynamic code dispatch depending on underlying ISA
> Test success criteria
> 
> 1.       All intrinsified VectorAPI methods are covered
> 
> 2.       All vector shapes and sizes combinations are covered
> 
> 3.       Test pass rate is 100% when tested on Skylake Server with default configuration

Ok, with the HotSpot options that should cover the various subsets of supported instructions on other kinds of server.


> 
> 4.       Test success stability is 99%

> Test methodology
> The test infrastructure consists of files and scripts which goal is to prepare and run the tests, and visualize the test results.
> 
> -          Template files factorize tests by type (Unary, Binary etc)
> 
> -          Shell script generates test bits for each vector shape and size
> 
> o   gen-template.sh details all the supported tests.
> 
> o   gen-tests.sh is used to generate all the tests files.
> 
> -          The test infrastructure uses jtreg to launch the tests
> 
> -          Each test:
> 
> o   Has a reference Java implementation of the Vector API feature being tested.
> 
> o   Takes functors as inputs used to mimics a set of realistic use-case scenario
> 
> o   Asserts that the result of the Vector API call matches the reference implementation
> 
> -          In case an assertion is failing, the test will generate an exception
> 
> o   Details about incorrect calculations or crashes will be available for later use
> Test Inventory
> Two Java source files are generated per vector shape automatically by a template mechanism.
> Example: Integers vectors, 128bits long:
> 
> -          Int128VectorTests.java: contains all API calls-related tests
> 
> -          Int128VectorLoadStoreTests.java: contains all Load/Store-related tests
> Test configurations
> Testing should cover:
> 
> -          Testing on both Linux and Windows

Mac too?


> 
> -          Testing with every combination of UseSSE=0 through UseSSE=4 and UseAVX=0 through UseAVX=3

Only need to test those combinations with -XX:+UseVectorApiIntrinsics?


> 
> -          Testing with -XX:+UseVectorApiIntrinsics and also with -XX:-UseVectorApiIntrinsics

The latter will ensure testing of the Java implementations.


It would be useful to mention that test execution time for one whole run under a single set of arguments is reasonable (e.g. < 2 minutes say) with possible improvements.

Thanks,
Paul.