Using JMH to test virtual thread performance

[Firouz Bharthania]

There reason to use JMH was to measure the performance of the following idea I have:

The CompletionPolicy.handle method is not thread safe and the user has to take that into his/her account. 
I can’t think of any scenario for which the developers wouldn’t need some sort of synchronization when implementing handle method because if what has to be taken care of is a task specific thing then it could have been done at the end of the task itself after it finishes. So whatever we want to implement in handle is in a way or another dealing with (partial)aggregation of results which would need some sort of synchronization.
In order to make it easier for developers, I was thinking of implementing a singe thread version of handle. It receives a SynchronousQueue from which it can take the results. With SynchronousQueue it wouldn’t consume any extra memory and also backpreasure would be automatically handled.

The handle method I’m thinking about would have a signature like "boolean handle(SynchronousQueue<Future> results)”. The returned boolean indicates if the whole scope has to shut down or not.

And in that regard I ran the following code to measure some performance but as I mentioned I get very high and unacceptable error margin.

Following is the code I use to measure virtual thread performance in case you are interested in:


@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Benchmark)
@OperationsPerInvocation(DifferentCompletionSpeedTestJMH.OPERATIONS_PER_INVOCATION)
public class DifferentCompletionSpeedTestJMH {
    public static final int OPERATIONS_PER_INVOCATION = 50_000_000;
    public static final int numberOfThreads = 1_000_000;

    public static void main(String... args) {
        final Options options = new OptionsBuilder()
                .include(DifferentCompletionSpeedTestJMH.class.getSimpleName())
                .forks(1)
                .build();
        try {
            new Runner(options).run();
        } catch (RunnerException e) {
            e.printStackTrace();
        }
    }

    @Benchmark
    public void usingNothing() throws InterruptedException{
        var latch = new CountDownLatch(1);
        var random = new Random();
        try(StructuredExecutor executor = StructuredExecutor.open()) {
            IntStream.range(0, numberOfThreads).forEach(a -> {
                executor.fork(() -> {
                            int rand = random.nextInt(1000);
//                            sleep(rand);
                            latch.await();
                            return rand;
                        }
                        , new StructuredExecutor.CompletionHandler<Integer>() {
                            @Override
                            public void handle(StructuredExecutor executor, Future<Integer> future) {

                            }
                        });
            });
            latch.countDown();
            executor.join();
        }
    }

    
    @Benchmark
    public void usingSynchronousQueue() throws InterruptedException{
        var synchQueue = new SynchronousQueue<Integer>();
        var list = new LinkedList<Integer>();
        var latch = new CountDownLatch(1);
        var random = new Random();
        try(StructuredExecutor executor = StructuredExecutor.open()) {
            IntStream.range(0, numberOfThreads).forEach(a -> {
                executor.fork(() -> {
                            int rand = random.nextInt(1000);
//                            sleep(rand);
                            latch.await();
                            return rand;
                        }
                        , new StructuredExecutor.CompletionHandler<Integer>() {
                            @Override
                            public void handle(StructuredExecutor executor, Future<Integer> future) {
                                try {
                                    synchQueue.put(future.get());
                                } catch (InterruptedException e) {
                                    e.printStackTrace();
                                } catch (ExecutionException e) {
                                    e.printStackTrace();
                                }
                            }
                        });
            });

            //single thread responsible for receiving all task results
            executor.fork(() -> {
                int c = 0;
                do {
                    list.add(synchQueue.take());
                }while(++c<numberOfThreads);
                return null;
            });
            latch.countDown();
            executor.join();
        }
    }


    static void sleep(int a) {
        try{
            Thread.sleep((long)a);
        }catch(Exception e) {
            e.printStackTrace();
        }
    }
}

Firouz



> On Feb 8, 2022, at 7:09 AM, Dr Heinz M. Kabutz <heinz at javaspecialists.eu> wrote:
> 
> Hi Alan and Firouz,
> 
> since virtual threads are currently executed on carrier threads from a FJP, we could re-purpose my recent newsletter to test the performance of any ForkJoinPool, not just the common pool:
> 
> https://www.javaspecialists.eu/archive/Issue297-Measuring-ForkJoinPool-Parallelism.html
> 
> This would not accurately measure the performance of a single virtual thread, but would give us a good indication of how the carrier thread are doing as a whole.
> 
> Regards
> 
> Heinz
> -- 
> Dr Heinz M. Kabutz (PhD CompSci)
> Author of "The Java™ Specialists' Newsletter" - www.javaspecialists.eu
> Java Champion - www.javachampions.org
> JavaOne Rock Star Speaker
> Tel: +30 69 75 595 262
> Skype: kabutz
> 
> On 2022/02/08 14:02, Alan Bateman wrote:
>> On 08/02/2022 00:22, Firouz Bharthania wrote:
>>> Hello,
>>> 
>>> Can JMH (Java Microbenchmarking Harness) be used for measuring virtual threads at this time, because I get high, unacceptable error margins when I run tests?
>>> 
>> Aleksey may have some advice but I would expect it should be possible to create a runner that invokes org.openjdk.jmh.Main.main in the context of a virtual thread. I don't know if anyone has done that.
>> 
>> -Alan