some confusion about bookkeeper performance

[zjflike]

QUESTION

I have tested two different bookkeeper, which is distributedlog-0.3.51-RC1 + bookkeeper 4.3.4-TWTTR and distributedlog-0.3.50 + bookkeeper 4.5.0. what confused me is that bookkeeper 4.5.0 has a worse performance compared with bookkeeper 4.3.4-TWTTR, and here is my test report:

in bookkeeper 4.3.4-TWTTR:

"bookkeeper_server.ADD_ENTRY" : {
"count" : 3425445,
"max" : 7.41E-4,
"mean" : 4.579571984435798E-5,
"min" : 3.0E-5,
"p50" : 4.4000000000000006E-5,
"p75" : 4.6E-5,
"p95" : 5.254999999999996E-5,
"p98" : 5.741999999999996E-5,
"p99" : 8.371000000000004E-5,
"p999" : 7.353740000000008E-4,
"stddev" : 3.2124531013854395E-5,
"m15_rate" : 835.1689420800385,
"m1_rate" : 648.1885926153338,
"m5_rate" : 826.3971430083111,
"mean_rate" : 495.75097584866603,
"duration_units" : "seconds",
"rate_units" : "calls/second"
}

in bookkeeper 4.5.0:

"bookkeeper_server.ADD_ENTRY" : {
"count" : 1989593,
"max" : 8.550560000000001E-4,
"mean" : 6.543760311284048E-5,
"min" : 3.1085000000000005E-5,
"p50" : 5.8833000000000004E-5,
"p75" : 6.157950000000001E-5,
"p95" : 7.241495E-5,
"p98" : 9.612595999999987E-5,
"p99" : 4.2929724000000005E-4,
"p999" : 8.536968280000003E-4,
"stddev" : 5.917626043631808E-5,
"m15_rate" : 830.11667426358,
"m1_rate" : 349.7475662441205,
"m5_rate" : 669.5704800826474,
"mean_rate" : 178.61990685543887,
"duration_units" : "seconds",
"rate_units" : "calls/second"
}

Addition:
I changed bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java line 548 to get less group time:
//this.maxGroupWaitInNanos = TimeUnit.MILLISECONDS.toNanos(conf.getJournalMaxGroupWaitMSec()); this.maxGroupWaitInNanos = 100;

The above results are tested from same computer with the same config(MSG_SIZE=256, MAX_RATE=40000, output-buffer-size=16384). My confusion is below:

1. Different bookkeeper shows different mean_rate under a same config. Maybe the new distributedlog 0.3.50 changes the rate of transmit package to achieve higher performance? Or bookkeeper 4.5.0 has new mechanism to deal with the package?

2. Compared with bookkeeper 4.3.4, bookkeeper 4.5.0 costs more 20us to deal with the request. Is this caused by different rate? Or what cause a worse performance?

Is there anyone meet this question? Can anyone explain that? It is troubled me for a long time.
Thank you very much for your time and patience !!! And forgive my pool English ^_^

[sijie]

I think the bk 4.5.0 (using v3 protocol) has slightly increased gc due to bytestring needs to copy memory. there is a change recently bumping the protobuf to 3.x, which will help the situation.

You might consider using latest DL master with 4.5.0, which it leverages the netty 4.x ByteBuf for managing the memory for output buffer. It would have better latency and throughput.

I will have a performance comparison report soon for describe the differences between versions.

[sijie]

Hope this explain your question.

[zjflike]

Firstly, thank you very much for your answer!!!
I am sorry that I made a mistake in my "QUESTION": Actually I used latest DL master with BK 4.5.0, and DL-0.3.51-RC1 with BK 4.3.4-TWTTR, and the results are above, shows that 4.5.0 costs more 20us to deal with the request. I don't know what cause this strange result, config ? Or something else...

[sijie]

@zjflike - I think the slight increased 20us is coming from the v3 (protobuf) protocol. using 2.x protobuf, there is no memory-copy-free method to serialize the bytestring into protobuf structure. I need to check more to confirm.