Creating a Blackhole

[Aleksey Shipilev]

Hi Andrew,

On 06/18/2015 07:17 PM, Andrew Brampton wrote:
> I have a complex benchmark, that takes >1 hour to run. I kept hitting an
> issue where the benchmark would crash during one of the setup/teardown
> phase (due to mistakes in my code). So I figured why don't I unit test my
> benchmark, to ensure the assumptions I make hold true.

I understand the use case. However, mocking Blackhole seems not to be
enough to test the generic JMH-driven benchmark? E.g. using other
@State-s would require mocking them as well, including special
{Benchmark,Iteration,Thread}Params, making sure all threads get their
relevant instance of state object, @Setup/@TearDown run in the same
order JMH does it, etc.

Given that, would it be easier to put the Asserts straight into
@Setup/@TearDown code, and run the benchmark with shorter data set? Our
entire integration testing suite is driven like that:

http://hg.openjdk.java.net/code-tools/jmh/file/1db18e73d21c/jmh-core-it/src/test/java/org/openjdk/jmh/it

> I can either do a nasty hack to create the Blackhole, or I'm requesting you
> add a new static method,  Blackhole.newBlackholeYesIdoKnowWhatImDoing(), or
> similar. Since in my case, I don't care about the benchmarking, and I'm
> calling this from a single junit thread, and just testing the correctness
> of my benchmark code. Alternatively if you create a Blackhole interface, I
> could easily create mock blackhole, for my testing.

Factory method lends itself for abuse: no matter how you call the
method, people would eventually call it in some weird scenario, without
reading the documentation.

Interface is out of the question: Blackhole.consume invocation
performance is very important.


> Suggestions?

a) (Boot-)classpath your own version of Blackhole class with proper
public constructor, or do some other kind of bytecode magic.

b) Ask us to provide a system property that disables the check in
Blackhole constructor. Can your testing environment set the Java options?

c) Ask us to ditch the exception from Blackhole constructor, and make
the constructor private/protected instead. Then you can circumvent the
protection offered for common use case by using Reflection with
setAccessible(true)?


> P.S For the curious, my code lives here https://github.com/bramp/unsafe
> under unsafe-benchmark.

404.

-Aleksey