Client reporter and metrics system

[xx789633]

The reporter component within the client program should collect and report all predefined metrics—like P99 latency, client memory utilization, and write QPS—either upon completion of the benchmarking process or during its execution. For this purpose, we can leverage the Codahale Metrics Toolkit (https://metrics.dropwizard.io/).

[polyzos]

Should we add a configurable data generation payload, either messages/sec or bytes/second? Or have it static?
For example, 10k messages/sec per writer or 10MB/second/writer?

[michaelkoepf]

@polyzos i think configurable workload in terms of messages/second makes sense. bytes/second can often be configured indirectly, if the underlying workload allows to configure the payload size.

independently of my comment above, two more things.

1. the workload generator should have a safe guard that validates if the data generation process can actually keep up with the configured rate (messages/second). why? because the data generation process may involve computationally more heavy parts, e.g., prng. hence, the workload generation process may not be able to sustain the configured rate. since we do not know what machine the generator will run on, this is necessary even if the rate is not configurable.
2. this comment is maybe better placed in the issues that implement the benchmarks, but: i would prefer to keep the configurable parameters to a minimum. e.g., apache beams nexmark implementation has a lot of knobs, but not all combinations of parameters are sensible. keeping configurable parameters to a minimum limits the risk of third parties using the suite to produce meaningless benchmark results and avoids drawing potentially false conclusions from the benchmarks.

[polyzos]

+1 on the above.. and the data generation check makes lots of sense, especially if we generate lots of columns (for testing projection pushdown - i.e, 50-100 columns), which might result in longer generation times and larger payloads

[michaelkoepf]

@xx789633 are there already details -- or even a poc -- how latency will be measured?

[xx789633]

Hi @michaelkoepf @polyzos , in my PoC code, the following parameters control the writing behavior for the insert test.

Item	Parameter	Description
Write configurations	threadSize	The number of threads used to generate data.
	testByTime	Specifies whether to perform the test at the specified point in time or based on the specified number of table rows.
	rowNumber	The number of table rows that you want to test. This parameter takes effect only when the value of the testByTime parameter is false.
	testTime	The time when you want to perform the test. Unit: milliseconds. This parameter takes effect only when the value of the testByTime parameter is true.
Table configurations	columnCount	The number of table columns. The data type of each column is STRING by default.
	columnSize	The length of data in each column of the table.
	tableName	The name of the table used for testing.

For upsert test, we have the writeColumnCount parameter. This parameter specifies the number of table columns where data is written.

[xx789633]

For the latency measurement, I use the meter and histogram toolkit from Java metrics library (https://metrics.dropwizard.io/4.2.0/getting-started.html). I just referenced Hologres benchmark program (https://github.com/aliyun/alibabacloud-hologres-connectors/blob/master/holo-e2e-performance-tool/src/main/java/com/alibaba/hologres/performace/client/ScanTest.java#L159)

[michaelkoepf]

@xx789633 thanks, this answers my question :) i was just interested if we are talking about the time it takes to ingest a record into fluss or the time it takes until a record becomes available in a downstream consumer that uses streaming read, e.g., flink. both are latency measurements, but with different notions.

[xx789633]

Hi @michaelkoepf @polyzos, please note that the configuration parameters I mentioned above are specifically tailored for ​primary-key table benchmarks. For log table benchmarks, we should indeed incorporate the "publish rate" parameter (defining the message production throughput in messages/sec from producers). I will review the OpenMessaging Benchmark Framework documentation and figure out how it measures pub/cons rate and latency for messaging systems.

[xx789633]

Hi @michaelkoepf @polyzos , regarding end-to-end latency measurement, I think the producer can attach a publish timestamp to the message. This approach would enable the consumer to calculate the end-to-end latency by referencing this timestamp.

For the data generation, we can define a Pub Delay metric that quantifies the delay in message production relative to the ideal throughput.