[Umbrella] Fluss-benchmarks Action Items

[xx789633]

The fluss-benchmarks repository provides a comprehensive suite of tools designed to simplify performance evaluation of Fluss clusters across diverse scenarios.

This ticket is an umbrella issue and is used to track action items.

Current Action Items:

• Client reporter and metrics system #4
• Primary-key table benchmarks (insert, upsert, point lookups) #3
• Log table benchmark (publishing/consuming throughput & latency) #2
• Filter pushdown performance for log table (FIP-10) #5
• Column pruning performance for log table #6

[polyzos]

A few questions:

1. What should be the base cluster setup for running these benchmarks?
2. Should we also test with lakehouse enabled?
3. Should we also have by default grafana/prometheus enabled for metrics? we could also collect some screenshots for these?
   WDTY?

[xx789633]

@polyzos

1. I think we can develop a flexible deployment script that allows users to easily set up Fluss clusters with arbitrary configurations of their choice.
2. Yes, we can include Lakehouse testing in our benchmarks. That said, I’m currently unsure about specific test cases or metrics to prioritize for the Lakehouse component.
3. For initial monitoring, I think we can start with foundational metrics such as P99 latency, client memory utilization, and write QPS. These can be directly reported by the client to keep overhead minimal. If more granular or complex metrics are needed later, we can integrate grafana/prometheus for enhanced visualization and analysis.

[polyzos]

Sounds great, and indeed, we can leave lakehouse tables out of scope for now, until we come up with some concrete use cases.

[platinumhamburg]

@polyzos

1. I think we can develop a flexible deployment script that allows users to easily set up Fluss clusters with arbitrary configurations of their choice.

I've encountered some great practices in other projects. In the Neon database project, they provide a neon_local binary tool in the control plane module that can easily set up and manage multiple local clusters with different configurations. This allows users and developers to quickly get started and enables end-to-end test suites implemented in pytest. In the Hologres project, they developed a nice tool called honebox (Hologres in a box) that supports quickly starting a containerized local cluster easily built from source code. This tool is also very convenient for development and end-to-end testing.