RFR: 8259074: regex benchmarks and tests

[Martin Buchholz]

On Mon, 1 Feb 2021 20:52:12 GMT, Martin Buchholz <martin at openjdk.org> wrote:

>> The assertion discussion aside, the micros look fine to me. 
>> 
>> With an eye towards reducing total run time I'd ask you to consider if all parameter combinations are useful or if we can get the same value after some pruning.
>
> @cl4es I agree pruning is a good idea. I settled on 3 data points with 16x separations as good enough to clearly show the difference between O(1) O(N) O(N^2) and O(2^N) (although O(2^N) would "run forever").
> 
> (although ... please tell me you're not actually running these benchmarks in an automated fashion ... too expensive, and needs a human to interpret the results)

> A manual exploration of a new set of micros would naturally start with
> the default config, so if such a config runs forever, that would be poor
> ergonomics IMHO. I don't think such configurations should be checked in
> in an active state.

We're actually in agreement.  There's no actual O(2^N) operation here, and jmh is similar to jtreg in having timeouts indicating failure.

Although I've been using shell loops as included in the class comments, I'll make sure running the tests using all the defaults gives sensible results.

I'm surprised to see jmh use the same number (5) of warmup and measurement iterations.  Unless you're looking for very small effects, one warmup run should be sufficient.

-------------

PR: https://git.openjdk.java.net/jdk/pull/1940