Skip to content

Commit 5c60ffa

Browse files
filipecosta90filipecosta90
committed
Initial commit
1 parent 2320ebf commit 5c60ffa

File tree

4 files changed

+216
-0
lines changed

4 files changed

+216
-0
lines changed

‎Readme.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
## Benchmark specifications goal
2+
3+
The Redis benchmarks specification describes the cross-language/tools requirements and expectations to foster performance and observability standards around redis related technologies.
4+
5+
Members from both industry and academia, including organizations and individuals are encouraged to contribute.
6+
7+
Currently, the following members actively support this project:
8+
9+
- [Redis Labs](https://redislabs.com/): providing steady-stable infrastructure platform to run the benchmark suite. Supporting the active development of this project with the company.
10+
11+
12+
## Directory layout
13+
14+
* `setups`
15+
* [`platforms`](./setups/platforms/): contains the standard platforms considered to provide steady stable results, and to represent common deployment targets.
16+
* [`topologies`](./setups/topologies/): contains the standard deployment topologies definition with the associated minimum specs to enable the topology definition.
17+
* [`test_suites`](./test-suites/): contains the benchmark suites definitions, specifying the target redis topology, the tested commands, the benchmark utility to use (the client), and if required the preloading dataset steps.
18+
* [`validator`](./validator/): contains the benchmark specifications validator utility
19+
* [`build_agent`](./build_agent/): contains the benchmark build agent utility that receives an event indicating a new build variant, generates the required redis binaries to test, and triggers the benchmark run on the listening agents.
20+
* [`benchmark_coordinator`](./benchmark_coordinator/): contains the coordinator utility that listens for benchmark suite run requests and setups the required steps to spin the actual benchmark topologies and to trigger the actual benchmarks.
21+
22+
23+
## Scope
24+
25+
This repo aims to provide Redis related benchmark standards and methodologies for:
26+
27+
- Management of benchmark data and specifications across different setups
28+
29+
- Running benchmarks and recording results
30+
31+
- Exporting performance results in several formats (CSV, RedisTimeSeries, JSON)
32+
33+
- **[SOON]** Finding on-cpu, off-cpu, io, and threading performance problems by attaching profiling tools/probers ( perf (a.k.a. perf_events), bpf tooling, vtune )
34+
35+
- **[SOON]** Finding performance problems by attaching telemetry probes
36+
37+
Current supported benchmark tools:
38+
39+
- [redis-benchmark](https://github.com/redis/redis)
40+
- [SOON][memtier_benchmark](https://github.com/RedisLabs/memtier_benchmark)
41+
- [SOON][redis-benchmark-go](https://github.com/filipecosta90/redis-benchmark-go)
42+
- [SOON][redis-benchmark-go](https://github.com/filipecosta90/redis-benchmark-go)
43+
44+
45+
# Contributing guidelines
46+
47+
## Adding new test suites
48+
49+
TBD
50+
51+
## Adding new topologies
52+
53+
TBD
54+
55+
## Adding new test platforms
56+
57+
TBD
58+
59+
# License
60+
61+
redis-benchmark-specifications is distributed under the Apache 2 license - see [LICENSE](LICENSE)

‎setups/platforms/aws-ec2-1node-c5.4xlarge.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
kind: VM
2+
3+
infrastructure_as_code:
4+
trigger_mode: "provision"
5+
provisioning:
6+
setup_teardown_required: true
7+
definition_language: "hcl"
8+
definition_contributor: redislabs
9+
definition: https://github.com/RedisLabsModules/testing-infrastructure/tree/master/terraform/oss-1node-c5.4xlarge
10+
units: 1
11+
provider: "aws"
12+
os: "ubuntu18.04"
13+
os_arch: "linux/amd64"
14+
user: "ubuntu"
15+
pem: "benchmarks.redislabs.pem"
16+
17+
metadata:
18+
labels:
19+
provider: "aws"
20+
instance_type: "c5.4xlarge"
21+
os: "ubuntu18.04"
22+
os_arch: "linux/amd64"
23+
definition_contributor: redislabs
24+
25+
unit-specs:
26+
cpu: 16
27+
memory: 32

‎setups/topologies.yml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
spec:
2+
setups:
3+
- name: oss-standalone
4+
type: oss-standalone
5+
redis_topology:
6+
primaries: 1
7+
replicas: 0
8+
resources:
9+
requests:
10+
cpu: "1000m"
11+
memory: "10g"
12+
limits:
13+
cpu: "2000m"
14+
memory: "10g"
15+
16+
- name: oss-standalone-1replica
17+
type: oss-standalone
18+
redis_topology:
19+
primaries: 1
20+
replicas: 1
21+
placement: "sparse"
22+
resources:
23+
requests:
24+
cpu: "2000m"
25+
memory: "10g"
26+
limits:
27+
cpu: "3000m"
28+
memory: "10g"
29+
30+
- name: oss-cluster-3-primaries
31+
type: oss-cluster
32+
redis_topology:
33+
primaries: 3
34+
replicas: 0
35+
placement: "sparse"
36+
resources:
37+
requests:
38+
cpu: "3000m"
39+
memory: "30g"
40+
limits:
41+
cpu: "6000m"
42+
memory: "30g"
43+
44+
- name: oss-cluster-5-primaries
45+
type: oss-cluster
46+
redis_topology:
47+
primaries: 5
48+
replicas: 0
49+
placement: "sparse"
50+
resources:
51+
requests:
52+
cpu: "5000m"
53+
memory: "50g"
54+
limits:
55+
cpu: "7000m"
56+
memory: "50g"
57+
58+
- name: oss-cluster-9-primaries
59+
type: oss-cluster
60+
redis_topology:
61+
primaries: 9
62+
replicas: 0
63+
placement: "sparse"
64+
resources:
65+
requests:
66+
cpu: "9000m"
67+
memory: "90g"
68+
limits:
69+
cpu: "10000m"
70+
memory: "90g"
71+
72+
- name: oss-cluster-15-primaries
73+
type: oss-cluster
74+
redis_topology:
75+
primaries: 15
76+
replicas: 0
77+
placement: "sparse"
78+
resources:
79+
requests:
80+
cpu: "15000m"
81+
memory: "150g"
82+
limits:
83+
cpu: "17000m"
84+
memory: "150g"
85+
86+
- name: oss-cluster-30-primaries
87+
type: oss-cluster
88+
redis_topology:
89+
primaries: 30
90+
replicas: 0
91+
placement: "sparse"
92+
resources:
93+
requests:
94+
cpu: "30000m"
95+
memory: "300g"
96+
limits:
97+
cpu: "32000m"
98+
memory: "300g"

‎test-suites/redis-benchmark-full-suite-1Mkeys-100B.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
version: 0.3
2+
name: "redis-benchmark-full-suite-1Mkeys-100B"
3+
description: "Runs the default redis-benchmark test suite, for a keyspace length of 1M keys
4+
with a data size of 100 Bytes for each key. On total 50 concurrent connections
5+
will be used, sending 5M requests."
6+
tested-commands:
7+
- PING
8+
- SET
9+
- GET
10+
- INCR
11+
- LPUSH
12+
- RPUSH
13+
- LPOP
14+
- RPOP
15+
- SADD
16+
- SPOP
17+
- ZADD
18+
- ZPOPMIN
19+
- LRANGE
20+
- MSET
21+
redis-topology: oss-standalone
22+
clientconfig:
23+
- tool: redis-benchmark
24+
- min-tool-version: "6.2.0"
25+
- parameters:
26+
- clients: 50
27+
- requests: 5000000
28+
- threads: 2
29+
- pipeline: 1
30+
- r: 1000000

0 commit comments

Comments
 (0)