Generation of list of benchmarks from IntelliJ IDEA

[Aleksey Shipilev]

Hi,

Still holidays here, so replying briefly to get you started.

On 01/05/2015 12:39 PM, Evgeny Mandrikov wrote:
> There is weird behavior when running benchmarks from IntelliJ IDEA: "No
> benchmarks to run; check the include/exclude regexps." is reported each
> time after modification of non-benchmark Java-files, but which are located
> in module with benchmarks. I can reproduce this with latest IntelliJ IDEA
> 14.0.2 and the same was with version 13. And this is a bit annoying.

> I didn't had time to investigate deeper, so just few quick questions:
> Have you encountered this problem before?

I've seen anecdotal reports about this on Eclipse.

> Do you think that root cause is in the way IntelliJ IDEA executes
> annotation processors, i.e. on their side? Or might it be that JMH violates
> some annotation processor constraints?

IIRC, the "problem" is that annotation processors can be called with the
subset of files at IDE's discretion, but JMH's BenchmarkGenerator will
overwrite the benchmark metadata list (META-INF/BenchmarkList) based on
current annotation processing session. Therefore, if the IDE saves the
non- at Benchmark file, JMH could end up with empty metadata file.

> In any case - do you think that JMH can be updated to prevent this? As an
> option I can imagine that existing file will be updated by JMH instead of
> overwrite. Do you see any obvious problem with such approach? If not, then
> I'm willing to work on patch.

Yes, appending the benchmark line to the metadata file would be an
(somewhat ugly) option, if javac Filer would allow us to do so. AFAIK,
it only allows us to "create" (possibly overwriting) the new resource.

The alternative might be to split the metadata file on per-benchmark
basis, so that it would be normally overwritten without messing with
anything else.

Thanks,
-Aleksey.