storage: use single batch find for a fetch #446
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjaakola-aiven
force-pushed
the
jjaakola-aiven-use-single-batch-find-on-fetch
branch
4 times, most recently
from
06062ae to
4dcecf1
Compare
The fetch path had two batch finds, one for the DelayedFetch.tryCompleteDiskless and seconds at DelayedFetch.onComplete. Collect the batches at tryCompleteDiskless and reuse at onComplete. This reduces the database queries to batch coordinator.
jjaakola-aiven
force-pushed
the
jjaakola-aiven-use-single-batch-find-on-fetch
branch
from
4dcecf1 to
5945ae6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The fetch path had two batch finds, one for the DelayedFetch.tryCompleteDiskless and seconds at DelayedFetch.onComplete.
Collect the batches at tryCompleteDiskless and reuse at onComplete. This reduces the database queries to batch coordinator.
The current code in main can have more batches at the onComplete as a new find batch query is run after the initial call at tryCompleteDiskless.
Comparisons to current main: 5875f03
On graphs the main is the left most run and right is from this PR.
Benchmark consumer fetch latency:
Find batches query time and rate:
Fetch rate:
