Benchmarking with heavily-assymetric threads? (full message)

[Lev Serebryakov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 19.11.2015 13:15, Ivan Krylov wrote:

 I'm using @Group("name") feature of JMH to start different threads in
one benchmark, I don't start threads by myself. But I'm blocking
benchmarked methods, as I try to benchmark different implementations
and strategies for blocking!

 Benchmark is rather simple (see at the end of message complete
example, you could not run it as you don't have my buffer
implementation, but it is straightforward *blocking* ring buffer for
single producer and single consumer).

 Now I'm understanding, that there is one major problem: if one of
group threads stops slightly before other one (perfectly possible),
other benchmark method could block forever (till JMH timeout), because
it will not have data to "consume" (or space in buffer to "produce").

@Fork(1)
@Warmup(iterations = 5, time = 10)
@Measurement(iterations = 10, time = 5)
@State(Scope.Group)
public class Buffers {
 @Param({"10", "512", "1024", "4096"})
 int WRITE_BATCH_SIZE;

 @Param({"1", "10", "512", "1024", "4096"})
 int READ_BATCH_SIZE;

 ByteBuffer buffer;
 int writeOaL[];
 int readOaL[];

 @Setup(Level.Iteration)
 public void Setup() {
  // Create ring buffer
  // It will have size which N*LCM(WRITE_BS, READ_BS),
  // where is N at least 2, but maybe more,
  // to have size greater than last argument (4096 in this case)
  buffer = (ByteBuffer)Buffer.getInstance(
    Buffer.TYPE.BYTE,
    WRITE_BATCH_SIZE, // Writes will be only in multiples of this
    READ_BATCH_SIZE,  // Reads will be only in multiples of this
    0,                // Not relevant for this benchmark
    4096              // Minimal size of buffer
  );

  // Offset and lengh of data to write, will be used many times
  writeOaL = new int[2];
  // Offset and lengh of data to read, will be used many times
  readOaL = new int[2];
 }

 @Benchmark
 @Group("buf")
 public void writer(Blackhole bh, Control ctl)
   throws InterruptedException {
  // Safeguard
  if (ctl.stopMeasurement)
   return;

  // Wait (block!) till buffer will have at least WRITE_BATCH_SIZE free
  // bytes.
  // writeOaL will be {writeOffset, availiableSpace} on return
  // (it doesn't used here)
  buffer.waitToWrite(WRITE_BATCH_SIZE, writeOaL);
  // Mark as-if we wrote WRITE_BATCH_SIZE bytes
  // Real byte copying is ommited! It is microbenchmark!
  buffer.finishToWrite(WRITE_BATCH_SIZE);
  // Avoid DCE, consume "length"
  bh.consume(writeOaL[1]);
 }

 @Benchmark
 @Group("buf")
 public void reader(Blackhole bh, Control ctl)
   throws InterruptedException {
  // Safeguard
  if (ctl.stopMeasurement)
   return;

  // Wait (block!) till buffer will have at least READ_BATCH_SIZE
  // bytes filled (availibale to read).
  // writeOaL will be {readOffset, availiableData} on return
  // (it doesn't used here)
  buffer.waitToRead(READ_BATCH_SIZE, readOaL);
  // Mark as-if we read READ_BATCH_SIZE bytes
  // Real byte copying is ommited! It is microbenchmark!
  buffer.finishToRead(READ_BATCH_SIZE);
  // Avoid DCE, consume "length"
  bh.consume(readOaL[1]);
 }
}

> Hm, would you mind sharing a sketch of you benchmark? Seems that
> you suggest the jmh changes thread priorities from what they would
> be without the harness. Do you start all threads from one benchmark
> method?
> 
> Thanks, Ivan
> 
> On 14/11/2015 22:14, Lev Serebryakov wrote:
>> On 14.11.2015 19:59, Lev Serebryakov wrote:
>> 
>>> For example, when imbalance is 1:10, one iteration takes when
>>> it is configured to be 10 seconds. When imbalance is 1:1000, I
>>> got *interrupted* iterations. Results looks meaningless, too.
>> Another problem is stopping. If Writer thread is stopped first,
>> here is possibility, that reader thread will block till timeout.
>> And I've got this problem in the wild :(
>> 
> 


- -- 
// Black Lion AKA Lev Serebryakov
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQJ8BAEBCgBmBQJWTca2XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF
QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EePcbMQAM0KewnS2OdnTPY9qy4zmEIv
cv+Xm76jJKjPASGVu+Jt5fsPmi0VLpbQGsHCcMzXmZvupb8uLS8shWFOolhVN+ss
6ZpNlWa5SSOtjD4skoKcYM2zWr8um9AkvqBi0SK8kRTX356a3kmSqJmvE+pIUzoy
HyYcdLPy6l8pM5XLOrEYVhraCyGuoKJCq5P/Bz80L3nSNrHO8Y/AjV3D+nhOGii5
o0zkLQiTDE5brqN69MKb07IJB+MESmRxquz68QrFiscl2AqP/93EC+oIWiMEpfg/
N51VzkktJ5PRd3MI6++npBDoB942X1spz88RShdvInbLjciQA5eOdWgBc/Yydeb6
1Hqbl76/PNPFOR9uEonuH8boW66JpE/0T02EbUbktphVQDcN7ErpxZUnwGDTfd7r
eFk8qksqUmrExKDWGCpPPm/TFkhxYOgreG3rXeveLGZs5Atas9PkTMJqEaqxxgCT
Gg5gzutseuoiIRJvCiFHR1y6+SrdwhVBQcyyYAMS0gAKwffBKFD6Ko5WdZvXTNX0
i6AtlYNrJOsY9hHPvU7C03hM04CKkYSwk8p7hrHmqMZaebCQpaEKlredfqB+kMr3
gHNhpuarmnkBGuJT4Kr/ZExJO38THV5pqf95x2Ib/DgS6rKMYILCfDNIgEHHjqMh
dIEO1bzir4/DaLFK8rCO
=3eBs
-----END PGP SIGNATURE-----