Skip to content

Commit 30c9ec3

Browse files
aldy505Copilot
andauthored
docs(self-hosted): more refined instructions for proper solution of kafka offset out of range error (#15741)
## DESCRIBE YOUR PR Avoid confusion around reseting Kafka offsets for folks that don't really touch Kafka in their life. ## IS YOUR CHANGE URGENT? Help us prioritize incoming PRs by letting us know when the change needs to go live. - [ ] Urgent deadline (GA date, etc.): <!-- ENTER DATE HERE --> - [ ] Other deadline: <!-- ENTER DATE HERE --> - [x] None: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've added an urgent due date to it. Thanks in advance for your help! ## PRE-MERGE CHECKLIST *Make sure you've checked the following before merging your changes:* - [ ] Checked Vercel preview for correctness, including links - [ ] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [ ] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) ## EXTRA RESOURCES - [Sentry Docs contributor guide](https://docs.sentry.io/contributing/) --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent f8c32a8 commit 30c9ec3

File tree

1 file changed

+18
-9
lines changed
  • develop-docs/self-hosted/troubleshooting

1 file changed

+18
-9
lines changed

develop-docs/self-hosted/troubleshooting/kafka.mdx

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -87,31 +87,41 @@ These solutions may result in data loss for the duration of your Kafka event ret
8787

8888
#### Proper solution
8989

90-
The _proper_ solution is as follows ([reported](https://github.com/getsentry/self-hosted/issues/478#issuecomment-666254392) by [@rmisyurev](https://github.com/rmisyurev)). This example uses `snuba-consumers` with `events` topic. Your consumer group name and topic name may be different.
90+
The _proper_ solution is as follows ([reported](https://github.com/getsentry/self-hosted/issues/478#issuecomment-666254392) by [@rmisyurev](https://github.com/rmisyurev)).
91+
92+
This example assumes you found the error from the `snuba-errors-consumer` container. Your consumer group name and topic name may be different.
9193

9294
1. Shutdown the corresponding Sentry/Snuba container that's using the consumer group (You can see the corresponding containers by inspecting the `docker-compose.yml` file):
95+
```yaml
96+
snuba-errors-consumer:
97+
<<: *snuba_defaults
98+
command: rust-consumer --storage errors --consumer-group snuba-consumers ...
99+
```
100+
According to the snippet above from `docker-compose.yml`, the consumer group is `snuba-consumers`. You need to find whether other containers are also using the same consumer group.
101+
In this case, `snuba-errors-consumer`, `snuba-outcomes-consumer`, `snuba-outcomes-billing-consumer`, `snuba-replays-consumer`, `snuba-profiling-profiles-consumer`,
102+
`snuba-profiling-functions-consumer` and `snuba-profiling-profile-chunks-consumer` are using the same consumer group, so you need to stop all of them:
93103
```shell
94-
docker compose stop snuba-errors-consumer snuba-outcomes-consumer snuba-outcomes-billing-consumer
104+
docker compose stop snuba-errors-consumer snuba-outcomes-consumer snuba-outcomes-billing-consumer snuba-replays-consumer snuba-profiling-profiles-consumer snuba-profiling-functions-consumer snuba-profiling-profile-chunks-consumer
95105
```
96-
2. Receive consumers list:
106+
2. Receive consumers list to make sure `snuba-consumers` is there:
97107
```shell
98108
docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --list
99109
```
100-
3. Get group info:
110+
3. Get group info for `snuba-consumers`. Here you will see the topics that the consumer group is subscribed to along with their partitions and current offset:
101111
```shell
102112
docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --describe
103113
```
104-
4. Watching what is going to happen with offset by using dry-run (optional):
114+
4. Watch what is going to happen with the offset by using dry-run (optional). This example uses the `events` topic found from the previous step:
105115
```shell
106116
docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --topic events --reset-offsets --to-latest --dry-run
107117
```
108-
5. Set offset to latest and execute:
118+
5. Set offset to latest and execute. Make sure to replace `events` with the topic you found from step 3:
109119
```shell
110120
docker compose exec kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --topic events --reset-offsets --to-latest --execute
111121
```
112122
6. Start the previously stopped Sentry/Snuba containers:
113123
```shell
114-
docker compose start snuba-errors-consumer snuba-outcomes-consumer snuba-outcomes-billing-consumer
124+
docker compose start snuba-errors-consumer snuba-outcomes-consumer snuba-outcomes-billing-consumer snuba-replays-consumer snuba-profiling-profiles-consumer snuba-profiling-functions-consumer snuba-profiling-profile-chunks-consumer
115125
```
116126
<Alert level="info" title="Tips">
117127
* You can replace <code>snuba-consumers</code> with other consumer groups or <code>events</code> with other topics when needed.
@@ -133,7 +143,7 @@ Unlike the proper solution, this involves resetting the offsets of all consumer
133143
#### Nuclear option
134144

135145
<Alert level="warning" title="Warning">
136-
The _nuclear option_ is removing all Kafka-related volumes and recreating them which _will_ cause data loss. Any data that was pending there will be gone upon deleting these volumes.
146+
The _nuclear option_ is to remove all Kafka-related volumes and recreate them. **This will cause data loss.** You'll lose any unprocessed events from the last 24 hours. Events that have already been processed and persisted in ClickHouse will remain safe.
137147
</Alert>
138148

139149
1. Stop the instance:
@@ -144,7 +154,6 @@ Unlike the proper solution, this involves resetting the offsets of all consumer
144154
```shell
145155
docker volume rm sentry-kafka
146156
```
147-
148157
3. Run the install script again:
149158
```shell
150159
./install.sh

0 commit comments

Comments
 (0)