Annotation API Fail?

[Aleksey Shipilev]

On 11/13/2017 11:59 AM, Jan Bernitt wrote:
> I'm new to JMH but if I don't totally miss something the annotation API
> design is ...well...awful.

First, you totally miss that coming to established project as outsider, and claim that things are
"awful" and "fail" is a bad introduction that would stain whatever legitimate points you have.

Second, the annotation API design is not really about "what anyone wants", but rather the common
denominator for features that have predictable default behavior, and cater for overwhelming majority
of cases. If users want anything more advanced, especially if that comes at odds with the static
nature of Java annotations, or other impedances -- including leaning to full-fledged DSLs -- they
are welcome to use the Java API.

Historically, @Benchmark was always the parameter-less marker annotation. Any new behavior was added
as separate annotation for simple reason: simplicity and compatibility. After exploring how to apply
JMH to large codebases, the metadata inheritance came into picture too: the ability to share the
benchmark configuration among the whole family of benchmarks trims down maintenance costs.

Suppose we do the single @Benchmark annotation. What does this example mean then?

@Benchmark(mode = Throughput, outputTimeUnit = SECONDS)
public class P {
}

public class Q extends P {

  @Benchmark(mode = AverageTime)
  public void k() {
  }

  public void l() {
  }
}

 - Is Q.l() a benchmark?
 - Does Q.k() has the time unit of SECONDS? Or is it default?
 - What about P.k() and P.l()? Are those benchmarks?

Also, as we talk about the implementation, how do you encode the default parameter for
outputTimeUnit so that implicit default value in @Benchmark in Q.k() does not override the explicit
value in P-class-scoped @Benchmark? Bonus points for solving this for non-reference parameters, e.g.
"int"-s.

Annotation inheritance is not alien for Java users, but annotation *attribute* inheritance --
especially the partial one -- is, so you also have the didactic problem.

> Didn't anyone want to name a benchmark something other than a method name?
> Method names are constrained. Why not allowing to override it?
> 
> @Benchmark(name="+")
> 
> There are a lot of cases where the operation we measure is much better
> summarised with a string that would be illegal as a method name.

The experience tells otherwise: in most cases, things that are measured as perfectly explained by
plain English, which coincidentally is alphanumeric, and is catered well by Java identifiers.
Deriving test names from method names seems natural because of that.

> Didn't anyone try to measure both average time and throughput for a method?
> Of course I want average time in another time unit than throughput.
> @OutputTimeUnit and @BenchmarkMode don't make much sense as separate
> annotations.

They kinda do, but historically @OTU was introduced much earlier than @BM, hence the disconnect.

> Then almost everything can be done with just the one @Benchmark annotation
> what is also much easier to document, comprehend and discover. And you can
> get rid of code.

Except for all the code you would need to write to handle @Benchmark merges and conflict resolution.
Also except for the costs of re-documentation, re-comprehension, and re-discovery by users. Unless
you can invest time into trying to create at least the prototype version of what you are suggesting,
it would stay "awful". Ain't nobody got time for API redesign.

-Aleksey