Skip to content

Commit 216e028

Browse files
markovamariamarkovamaria
authored
Added execution and updated installation section to Readme (#175)
* Added execution and updated installation section * update for --test option * Added new line sign * Moved Install and Execute sections to Contribution part * Added page's navigation * Removed Installation itself from installing packages dependencies * Up Install and Execution section
1 parent faa89ec commit 216e028

File tree

1 file changed

+157
-81
lines changed

1 file changed

+157
-81
lines changed

‎Readme.md‎

Lines changed: 157 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,31 @@
55
[![CI tests](https://github.com/redis/redis-benchmarks-specification/actions/workflows/tox.yml/badge.svg)](https://github.com/redis/redis-benchmarks-specification/actions/workflows/tox.yml)
66
[![PyPI version](https://badge.fury.io/py/redis-benchmarks-specification.svg)](https://pypi.org/project/redis-benchmarks-specification)
77

8-
9-
## Installation
10-
11-
To have access to the latest SPEC and Tooling impletamtion you only need to install one python package.
12-
13-
Installation is done using pip, the package installer for Python, in the following manner:
14-
15-
```bash
16-
python3 -m pip install redis-benchmarks-specification --ignore-installed PyYAML
17-
```
8+
<!-- toc -->
9+
10+
- [Benchmark specifications goal](#benchmark-specifications-goal)
11+
- [Scope](#scope)
12+
- [Installation and Execution](#installation-and-execution)
13+
- [Installing package requirements](#installing-package-requirements)
14+
- [Installing Redis benchmarks specification](#installing-redis-benchmarks-specification-implementations)
15+
- [Testing out the redis-benchmarks-spec-runner](#testing-out-the-redis-benchmarks-spec-runner)
16+
- [Testing out redis-benchmarks-spec-sc-coordinator](#testing-out-redis-benchmarks-spec-sc-coordinator)
17+
- [Architecture diagram](#architecture-diagram)
18+
- [Directory layout](#directory-layout)
19+
- [Specifications](#specifications)
20+
- [Spec tool implementations](#spec-tool-implementations)
21+
- [Contributing guidelines](#contributing-guidelines)
22+
- [Joining the performance initiative and adding a continuous benchmark platform](#joining-the-performance-initiative-and-adding-a-continuous-benchmark-platform)
23+
- [Joining the performance initiative](#joining-the-performance-initiative)
24+
- [Adding a continuous benchmark platform](#adding-a-continuous-benchmark-platform)
25+
- [Adding redis-benchmarks-spec-sc-coordinator to supervisord](#adding-redis-benchmarks-spec-sc-coordinator-to-supervisord)
26+
- [Development](#development)
27+
- [Running formaters](#running-formaters)
28+
- [Running linters](#running-linters)
29+
- [Running tests](#running-tests)
30+
- [License](#license)
31+
32+
<!-- tocstop -->
1833

1934

2035
## Benchmark specifications goal
@@ -54,100 +69,93 @@ Current supported benchmark tools:
5469
- [SOON][redis-benchmark-go](https://github.com/filipecosta90/redis-benchmark-go)
5570

5671

57-
## Installing Redis benchmarks specification implementations
72+
## Installation and Execution
5873

5974
The Redis benchmarks specification and implementations is developed for Unix and is actively tested on it.
75+
To have access to the latest SPEC and Tooling impletamtion you only need to install one python package.<br />
76+
Before package's installation, please install its' dependencies.
6077

61-
Installation is done using pip, the package installer for Python, in the following manner:
78+
### Installing package requirements
6279

6380
```bash
64-
python3 -m pip install redis-benchmarks-specification
65-
```
66-
67-
## Architecture diagram
68-
69-
![Architecture diagram](./arch-diagram.png)
70-
71-
In a very brief description, github.com/redis/redis upstream changes trigger an HTTP API call containing the
72-
relevant git information.
73-
74-
The HTTP request is then converted into an event ( tracked within redis ) that will trigger multiple build variants requests based upon the distinct platforms described in [`platforms`](redis_benchmarks_specification/setups/platforms/).
75-
76-
As soon as a new build variant request is received, the build agent ([`redis-benchmarks-spec-builder`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__builder__/))
77-
prepares the artifact(s) and proceeds into adding an artifact benchmark event so that the benchmark coordinator ([`redis-benchmarks-spec-sc-coordinator`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__self_contained_coordinator__/)) can deploy/manage the required infrastructure and DB topologies, run the benchmark, and export the performance results.
78-
## Directory layout
79-
80-
### Specifications
81-
82-
The following is a high level status report for currently available specs.
83-
84-
* `redis_benchmarks_specification`
85-
* [`test-suites`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/test-suites/): contains the benchmark suites definitions, specifying the target redis topology, the tested commands, the benchmark utility to use (the client), and if required the preloading dataset steps.
86-
87-
* `redis_benchmarks_specification/setups`
88-
* [`platforms`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/setups/platforms/): contains the standard platforms considered to provide steady stable results, and to represent common deployment targets.
89-
* [`topologies`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/setups/topologies/): contains the standard deployment topologies definition with the associated minimum specs to enable the topology definition.
90-
* [`builders`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/setups/builders/): contains the build environment variations, that enable to build Redis with different compilers, compiler flags, libraries, etc...
91-
92-
### Spec tool implementations
93-
94-
The following is a high level status report for currently available spec implementations.
95-
96-
* **STATUS: Experimental** [`redis-benchmarks-spec-api`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__api__/) : contains the API that translates the POST HTTP request that was triggered by github.com/redis/redis upstream changes, and fetches the relevant git/source info and coverts it into an event ( tracked within redis ).
97-
98-
* **STATUS: Experimental** [`redis-benchmarks-spec-builder`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__builder__/): contains the benchmark build agent utility that receives an event indicating a new build variant, generates the required redis binaries to test, and triggers the benchmark run on the listening agents.
99-
100-
* **STATUS: Experimental** [`redis-benchmarks-spec-sc-coordinator`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__self_contained_coordinator__/): contains the coordinator utility that listens for benchmark suite run requests and setups the required steps to spin the actual benchmark topologies and to trigger the actual benchmarks.
101-
102-
* **STATUS: Experimental** [`redis-benchmarks-spec-client-runner`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__runner__/): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.
103-
104-
## Contributing guidelines
105-
106-
### Adding new test suites
81+
# install pip installer for python3
82+
sudo apt install python3-pip -y
83+
sudo pip3 install --upgrade pip
10784

108-
TBD
85+
# install docker
86+
sudo apt install docker.io -y
10987

110-
### Adding new topologies
88+
# install supervisord
89+
sudo apt install supervisor -y
90+
```
11191

112-
TBD
11392

114-
### Joining the performance initiative and adding a continuous benchmark platform
93+
### Installing Redis benchmarks specification
11594

116-
#### Joining the performance initiative
117-
In order to join the performance initiative the only requirement is that you provide a steady-stable infrastructure
118-
platform to run the benchmark suites, and you reach out to one of the Redis Performance Initiative member via
119-
`performance <at> redis <dot> com` so that we can provide you with the required secrets to actively listen for benchmark events.
95+
Installation is done using pip, the package installer for Python, in the following manner:
12096

121-
If you check the above "Architecture diagram", this means you only need to run the last moving part of the arch, meaning you will have
122-
one or more benchmark coordinator machines actively running benchmarks and pushing the results back to our datasink.
97+
```bash
98+
python3 -m pip install redis-benchmarks-specification --ignore-installed PyYAML
99+
```
123100

124-
#### Adding a continuous benchmark platform
101+
To run particular version - use its number, e.g. 0.1.57:
102+
```bash
103+
pip3 install redis-benchmarks-specification==0.1.57
104+
```
125105

126-
In order to be able to run the benchmarks on the platform you need pip installer for python3, and docker.
127-
Apart from it, we recommend you manage the `redis-benchmarks-spec-sc-coordinator` process(es) state via a process monitoring tool like
128-
supervisorctl, lauchd, daemon tools, or other.
129106

130-
For this example we relly uppon `supervisorctl` for process managing.
107+
### Testing out the redis-benchmarks-spec-client-runner
131108

132-
##### Installing package requirements
109+
There is an option to run "redis-benchmarks-spec" tests using standalone runner approach. For this option redis-benchmarks-specificaiton should be run together with redis-server in the same time.
133110

134111
```bash
135-
# install pip installer for python3
136-
sudo apt install python3-pip -y
137-
sudo pip3 install --upgrade pip
112+
# Run redis server
113+
./src/redis-server --port 6379 --dir logs --logfile server.log --save "" [--daemonize yes]
138114

139-
# install docker
140-
sudo apt install docker.io -y
115+
# Run benchmark
116+
redis-benchmarks-spec-client-runner --db_server_host localhost --db_server_port 6379 --client_aggregated_results_folder ./test
117+
```
141118

142-
# install supervisord
143-
sudo apt install supervisor -y
119+
Option "--daemonize yes" given to server run command allows to run redis-server in background.<br />
120+
Option "--test X.yml" given to benchmark execution command allows to run particular test, where X - test name
144121

145-
# install benchmark specs
146-
python3 -m pip install redis-benchmarks-specification --ignore-installed PyYAML
122+
Full list of option can be taken with "-h" option:
123+
```
124+
$ redis-benchmarks-spec-client-runner -h
125+
usage: redis-benchmarks-spec-client-runner [-h]
126+
[--platform-name PLATFORM_NAME]
127+
[--triggering_env TRIGGERING_ENV]
128+
[--setup_type SETUP_TYPE]
129+
[--github_repo GITHUB_REPO]
130+
[--github_org GITHUB_ORG]
131+
[--github_version GITHUB_VERSION]
132+
[--logname LOGNAME]
133+
[--test-suites-folder TEST_SUITES_FOLDER]
134+
[--test TEST]
135+
[--db_server_host DB_SERVER_HOST]
136+
[--db_server_port DB_SERVER_PORT]
137+
[--cpuset_start_pos CPUSET_START_POS]
138+
[--datasink_redistimeseries_host DATASINK_REDISTIMESERIES_HOST]
139+
[--datasink_redistimeseries_port DATASINK_REDISTIMESERIES_PORT]
140+
[--datasink_redistimeseries_pass DATASINK_REDISTIMESERIES_PASS]
141+
[--datasink_redistimeseries_user DATASINK_REDISTIMESERIES_USER]
142+
[--datasink_push_results_redistimeseries] [--profilers PROFILERS]
143+
[--enable-profilers] [--flushall_on_every_test_start]
144+
[--flushall_on_every_test_end]
145+
[--preserve_temporary_client_dirs]
146+
[--client_aggregated_results_folder CLIENT_AGGREGATED_RESULTS_FOLDER]
147+
[--tls]
148+
[--tls-skip-verify]
149+
[--cert CERT]
150+
[--key KEY]
151+
[--cacert CACERT]
152+
redis-benchmarks-spec-client-runner (solely client) 0.1.61
153+
...
147154
```
148155

156+
### Testing out redis-benchmarks-spec-sc-coordinator
149157

150-
##### Testing out redis-benchmarks-spec-sc-coordinator
158+
Alternative way of running redis-server for listeting is running via redis-benchmarks coordinator.
151159

152160
You should now be able to print the following installed benchmark runner help:
153161

@@ -237,6 +245,74 @@ $ poetry run redis-benchmarks-spec-sc-coordinator --platform-name example-platfo
237245

238246
You're now actively listening for benchmarks requests to Redis!
239247

248+
249+
250+
## Architecture diagram
251+
252+
![Architecture diagram](./arch-diagram.png)
253+
254+
In a very brief description, github.com/redis/redis upstream changes trigger an HTTP API call containing the
255+
relevant git information.
256+
257+
The HTTP request is then converted into an event ( tracked within redis ) that will trigger multiple build variants requests based upon the distinct platforms described in [`platforms`](redis_benchmarks_specification/setups/platforms/).
258+
259+
As soon as a new build variant request is received, the build agent ([`redis-benchmarks-spec-builder`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__builder__/))
260+
prepares the artifact(s) and proceeds into adding an artifact benchmark event so that the benchmark coordinator ([`redis-benchmarks-spec-sc-coordinator`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__self_contained_coordinator__/)) can deploy/manage the required infrastructure and DB topologies, run the benchmark, and export the performance results.
261+
## Directory layout
262+
263+
### Specifications
264+
265+
The following is a high level status report for currently available specs.
266+
267+
* `redis_benchmarks_specification`
268+
* [`test-suites`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/test-suites/): contains the benchmark suites definitions, specifying the target redis topology, the tested commands, the benchmark utility to use (the client), and if required the preloading dataset steps.
269+
270+
* `redis_benchmarks_specification/setups`
271+
* [`platforms`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/setups/platforms/): contains the standard platforms considered to provide steady stable results, and to represent common deployment targets.
272+
* [`topologies`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/setups/topologies/): contains the standard deployment topologies definition with the associated minimum specs to enable the topology definition.
273+
* [`builders`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/setups/builders/): contains the build environment variations, that enable to build Redis with different compilers, compiler flags, libraries, etc...
274+
275+
### Spec tool implementations
276+
277+
The following is a high level status report for currently available spec implementations.
278+
279+
* **STATUS: Experimental** [`redis-benchmarks-spec-api`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__api__/) : contains the API that translates the POST HTTP request that was triggered by github.com/redis/redis upstream changes, and fetches the relevant git/source info and coverts it into an event ( tracked within redis ).
280+
281+
* **STATUS: Experimental** [`redis-benchmarks-spec-builder`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__builder__/): contains the benchmark build agent utility that receives an event indicating a new build variant, generates the required redis binaries to test, and triggers the benchmark run on the listening agents.
282+
283+
* **STATUS: Experimental** [`redis-benchmarks-spec-sc-coordinator`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__self_contained_coordinator__/): contains the coordinator utility that listens for benchmark suite run requests and setups the required steps to spin the actual benchmark topologies and to trigger the actual benchmarks.
284+
285+
* **STATUS: Experimental** [`redis-benchmarks-spec-client-runner`](https://github.com/filipecosta90/redis-benchmarks-specification/tree/main/redis_benchmarks_specification/__runner__/): contains the client utility that triggers the actual benchmarks against an endpoint provided. This tool is setup agnostic and expects the DB to be properly spinned beforehand.
286+
287+
## Contributing guidelines
288+
289+
### Adding new test suites
290+
291+
TBD
292+
293+
### Adding new topologies
294+
295+
TBD
296+
297+
### Joining the performance initiative and adding a continuous benchmark platform
298+
299+
#### Joining the performance initiative
300+
In order to join the performance initiative the only requirement is that you provide a steady-stable infrastructure
301+
platform to run the benchmark suites, and you reach out to one of the Redis Performance Initiative member via
302+
`performance <at> redis <dot> com` so that we can provide you with the required secrets to actively listen for benchmark events.
303+
304+
If you check the above "Architecture diagram", this means you only need to run the last moving part of the arch, meaning you will have
305+
one or more benchmark coordinator machines actively running benchmarks and pushing the results back to our datasink.
306+
307+
#### Adding a continuous benchmark platform
308+
309+
In order to be able to run the benchmarks on the platform you need pip installer for python3, and docker.
310+
Apart from it, we recommend you manage the `redis-benchmarks-spec-sc-coordinator` process(es) state via a process monitoring tool like
311+
supervisorctl, lauchd, daemon tools, or other.
312+
313+
For this example we relly uppon `supervisorctl` for process managing.
314+
315+
240316
##### Adding redis-benchmarks-spec-sc-coordinator to supervisord
241317

242318
Let's add a supervisord entry as follow

0 commit comments

Comments
 (0)