RFR: 8259074: regex benchmarks and tests

[Martin Buchholz]

On Tue, 2 Feb 2021 01:40:09 GMT, Martin Buchholz <martin at openjdk.org> wrote:

>> @cl4es I agree pruning is a good idea. I settled on 3 data points with 16x separations as good enough to clearly show the difference between O(1) O(N) O(N^2) and O(2^N) (although O(2^N) would "run forever").
>> 
>> (although ... please tell me you're not actually running these benchmarks in an automated fashion ... too expensive, and needs a human to interpret the results)
>
>> A manual exploration of a new set of micros would naturally start with
>> the default config, so if such a config runs forever, that would be poor
>> ergonomics IMHO. I don't think such configurations should be checked in
>> in an active state.
> 
> We're actually in agreement.  There's no actual O(2^N) operation here, and jmh is similar to jtreg in having timeouts indicating failure.
> 
> Although I've been using shell loops as included in the class comments, I'll make sure running the tests using all the defaults gives sensible results.
> 
> I'm surprised to see jmh use the same number (5) of warmup and measurement iterations.  Unless you're looking for very small effects, one warmup run should be sufficient.

I added annotations for sensible (faster) default, including @Warmup(iterations = 1)

-------------

PR: https://git.openjdk.java.net/jdk/pull/1940