Skip to content

Commit 73296e3

Browse files
simplestephsimplesteph
committed
first version
1 parent 6972ddf commit 73296e3

File tree

6 files changed

+339
-2
lines changed

6 files changed

+339
-2
lines changed

‎README.md‎

Lines changed: 104 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,104 @@
1-
# kafka-stack-docker
2-
docker compose files to create a fully working kafka stack
1+
# kafka-stack-docker-compose
2+
3+
This replicates as well as possible real deployment configurations, where you have your zookeeper servers and kafka servers actually all distinct from each other. This solves all the networking hurdles that comes with docker-compose, and is compatible cross platform. It only requires an update to your host files.
4+
5+
## Stack version
6+
7+
- Zookeeper version: 3.4.9
8+
- Kafka version: 0.10.1.0
9+
10+
## Single Zookeeper / Single Kafka
11+
12+
This configuration fits most development requirements.
13+
14+
- Zookeeper will be available at `localhost:2181`
15+
- Kafka will be available at `localhost:9092`
16+
17+
Make sure to add to your `/etc/hosts` file
18+
```
19+
127.0.0.1 kafka1
20+
```
21+
22+
Run with:
23+
```
24+
docker-compose -f zk-single-kafka-single.yml up
25+
docker-compose -f zk-single-kafka-single.yml down
26+
```
27+
28+
## Single Zookeeper / Multiple Kafka
29+
30+
If you want to have two brokers and experiment with replication / fault-tolerance.
31+
32+
- Zookeeper will be available at `localhost:2181`
33+
- Kafka will be available at `localhost:9092,localhost:9093`
34+
35+
Make sure to add to your `/etc/hosts` file
36+
```
37+
127.0.0.1 kafka1
38+
127.0.0.1 kafka2
39+
```
40+
41+
Run with:
42+
```
43+
docker-compose -f zk-single-kafka-multiple.yml up
44+
docker-compose -f zk-single-kafka-multiple.yml down
45+
```
46+
47+
## Multiple Zookeeper / Single Kafka
48+
49+
If you want to have three zookeeper and experiment with zookeeper fault-tolerance.
50+
51+
- Zookeeper will be available at `localhost:2181,localhost:2182,localhost:2183`
52+
- Kafka will be available at `localhost:9092`
53+
54+
Make sure to add to your `/etc/hosts` file
55+
```
56+
127.0.0.1 kafka1
57+
```
58+
59+
Run with:
60+
```
61+
docker-compose -f zk-multiple-kafka-single.yml up
62+
docker-compose -f zk-multiple-kafka-single.yml down
63+
```
64+
65+
66+
## Multiple Zookeeper / Multiple Kafka
67+
68+
If you want to have three zookeeper and two kafka brokers to experiment with production setup.
69+
70+
- Zookeeper will be available at `localhost:2181,localhost:2182,localhost:2183`
71+
- Kafka will be available at `localhost:9092,localhost:9093`
72+
73+
Make sure to add to your `/etc/hosts` file
74+
```
75+
127.0.0.1 kafka1
76+
127.0.0.1 kafka2
77+
```
78+
79+
Run with:
80+
```
81+
docker-compose -f zk-multiple-kafka-multiple.yml up
82+
docker-compose -f zk-multiple-kafka-multiple.yml down
83+
```
84+
85+
86+
## Full stack
87+
88+
- Single Zookeeper: `localhost:2181`
89+
- Single Kafka: `localhost:9092`
90+
- Kafka Schema Registry: `localhost:8081`
91+
- Kafka Rest Proxy: `localhost:8082`
92+
- Kafka Topics UI: `localhost:8000`
93+
94+
95+
Make sure to add to your `/etc/hosts` file
96+
```
97+
127.0.0.1 kafka1
98+
```
99+
100+
Run with:
101+
```
102+
docker-compose -f full-stack.yml up
103+
docker-compose -f full-stack.yml down
104+
```

‎full-stack.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
version: '2'
2+
3+
services:
4+
zoo1:
5+
image: zookeeper:3.4.9
6+
restart: always
7+
ports:
8+
- "2181:2181"
9+
environment:
10+
ZOO_MY_ID: 1
11+
ZOO_PORT: 2181
12+
ZOO_SERVERS: server.1=zoo1:2888:3888
13+
14+
kafka1:
15+
image: wurstmeister/kafka:0.10.1.0
16+
ports:
17+
- "9092:9092"
18+
environment:
19+
# add the entry "127.0.0.1 kafka1" to your /etc/hosts file
20+
KAFKA_ADVERTISED_HOST_NAME: "kafka1"
21+
KAFKA_ADVERTISED_PORT: "9092"
22+
KAFKA_PORT: "9092"
23+
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
24+
depends_on:
25+
- zoo1
26+
27+
kafka-schema-registry:
28+
image: confluentinc/cp-schema-registry
29+
ports:
30+
- "8081:8081"
31+
environment:
32+
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zoo1:2181
33+
SCHEMA_REGISTRY_HOST_NAME: kafka-schema-registry
34+
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
35+
depends_on:
36+
- zoo1
37+
- kafka1
38+
39+
kafka-rest-proxy:
40+
image: confluentinc/cp-kafka-rest:3.1.1
41+
ports:
42+
- "8082:8082"
43+
environment:
44+
KAFKA_REST_ZOOKEEPER_CONNECT: zoo1:2181
45+
KAFKA_REST_LISTENERS: http://0.0.0.0:8082/
46+
KAFKA_REST_SCHEMA_REGISTRY_URL: http://kafka-schema-registry:8081/
47+
KAFKA_REST_HOST_NAME: kafka-rest-proxy
48+
depends_on:
49+
- zoo1
50+
- kafka1
51+
- kafka-schema-registry
52+
53+
kafka-topics-ui:
54+
image: landoop/kafka-topics-ui
55+
ports:
56+
- "8000:8000"
57+
environment:
58+
KAFKA_REST_PROXY_URL: "http://kafka-rest-proxy:8082/"
59+
PROXY: "true"
60+
depends_on:
61+
- zoo1
62+
- kafka1
63+
- kafka-schema-registry
64+
- kafka-rest-proxy

‎zk-multiple-kafka-multiple.yml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
version: '2'
2+
3+
services:
4+
zoo1:
5+
image: zookeeper:3.4.9
6+
restart: always
7+
ports:
8+
- "2181:2181"
9+
environment:
10+
ZOO_MY_ID: 1
11+
ZOO_PORT: 2181
12+
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
13+
zoo2:
14+
image: zookeeper:3.4.9
15+
restart: always
16+
ports:
17+
- "2182:2182"
18+
environment:
19+
ZOO_MY_ID: 2
20+
ZOO_PORT: 2182
21+
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
22+
23+
zoo3:
24+
image: zookeeper:3.4.9
25+
restart: always
26+
ports:
27+
- "2183:2183"
28+
environment:
29+
ZOO_MY_ID: 3
30+
ZOO_PORT: 2183
31+
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
32+
33+
kafka1:
34+
image: wurstmeister/kafka:0.10.1.0
35+
ports:
36+
- "9092:9092"
37+
environment:
38+
# add the entry "127.0.0.1 kafka1" to your /etc/hosts file
39+
KAFKA_ADVERTISED_HOST_NAME: "kafka1"
40+
KAFKA_ADVERTISED_PORT: "9092"
41+
KAFKA_PORT: "9092"
42+
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
43+
depends_on:
44+
- zoo1
45+
- zoo2
46+
- zoo3
47+
48+
49+
kafka2:
50+
image: wurstmeister/kafka:0.10.1.0
51+
ports:
52+
- "9093:9093"
53+
environment:
54+
# add the entry "127.0.0.1 kafka2" to your /etc/hosts file
55+
KAFKA_ADVERTISED_HOST_NAME: "kafka2"
56+
KAFKA_ADVERTISED_PORT: "9093"
57+
KAFKA_PORT: "9093"
58+
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
59+
depends_on:
60+
- zoo1
61+
- zoo2
62+
- zoo3

‎zk-multiple-kafka-single.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
version: '2'
2+
3+
services:
4+
zoo1:
5+
image: zookeeper:3.4.9
6+
restart: always
7+
ports:
8+
- "2181:2181"
9+
environment:
10+
ZOO_MY_ID: 1
11+
ZOO_PORT: 2181
12+
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
13+
zoo2:
14+
image: zookeeper:3.4.9
15+
restart: always
16+
ports:
17+
- "2182:2182"
18+
environment:
19+
ZOO_MY_ID: 2
20+
ZOO_PORT: 2182
21+
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
22+
23+
zoo3:
24+
image: zookeeper:3.4.9
25+
restart: always
26+
ports:
27+
- "2183:2183"
28+
environment:
29+
ZOO_MY_ID: 3
30+
ZOO_PORT: 2183
31+
ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
32+
33+
kafka1:
34+
image: wurstmeister/kafka:0.10.1.0
35+
ports:
36+
- "9092:9092"
37+
environment:
38+
# add the entry "127.0.0.1 kafka1" to your /etc/hosts file
39+
KAFKA_ADVERTISED_HOST_NAME: "kafka1"
40+
KAFKA_ADVERTISED_PORT: "9092"
41+
KAFKA_PORT: "9092"
42+
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
43+
depends_on:
44+
- zoo1
45+
- zoo2
46+
- zoo3

‎zk-single-kafka-multiple.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
version: '2'
2+
3+
services:
4+
zoo1:
5+
image: zookeeper:3.4.9
6+
restart: always
7+
ports:
8+
- "2181:2181"
9+
environment:
10+
ZOO_MY_ID: 1
11+
ZOO_PORT: 2181
12+
ZOO_SERVERS: server.1=zoo1:2888:3888
13+
14+
kafka1:
15+
image: wurstmeister/kafka:0.10.1.0
16+
ports:
17+
- "9092:9092"
18+
environment:
19+
# add the entry "127.0.0.1 kafka1" to your /etc/hosts file
20+
KAFKA_ADVERTISED_HOST_NAME: "kafka1"
21+
KAFKA_ADVERTISED_PORT: "9092"
22+
KAFKA_PORT: "9092"
23+
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
24+
depends_on:
25+
- zoo1
26+
27+
kafka2:
28+
image: wurstmeister/kafka:0.10.1.0
29+
ports:
30+
- "9093:9093"
31+
environment:
32+
# add the entry "127.0.0.1 kafka2" to your /etc/hosts file
33+
KAFKA_ADVERTISED_HOST_NAME: "kafka2"
34+
KAFKA_ADVERTISED_PORT: "9093"
35+
KAFKA_PORT: "9093"
36+
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
37+
depends_on:
38+
- zoo1

‎zk-single-kafka-single.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
version: '2'
2+
3+
services:
4+
zoo1:
5+
image: zookeeper:3.4.9
6+
restart: always
7+
ports:
8+
- "2181:2181"
9+
environment:
10+
ZOO_MY_ID: 1
11+
ZOO_PORT: 2181
12+
ZOO_SERVERS: server.1=zoo1:2888:3888
13+
14+
kafka1:
15+
image: wurstmeister/kafka:0.10.1.0
16+
ports:
17+
- "9092:9092"
18+
environment:
19+
# add the entry "127.0.0.1 kafka1" to your /etc/hosts file
20+
KAFKA_ADVERTISED_HOST_NAME: "kafka1"
21+
KAFKA_ADVERTISED_PORT: "9092"
22+
KAFKA_PORT: "9092"
23+
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
24+
depends_on:
25+
- zoo1

0 commit comments

Comments
 (0)