[WIP] QUIC server #7
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds a QUIC-based implementation of file upload and download that uploads and downloads files on streams, thus allowing both parallel downloads/uploads, as well as allowing more efficient storage as streamed data can be sent piece-by-piece as it loads from disk, rather than needing large all-at-once reads and writes. This implementation makes heavy use of io_uring for efficiency, and as such will never work outside Linux. Each upload or download uses exactly one stream (skipping the first stream): an upload is prefixed with a bt-encoded file metadata block (containing the "PUT" request type, size, and optional ttl), while a download request consists of the same sort of block but containing "GET" and the file id. For PUT, all data on the stream after the initial block is the file data, terminated by a FIN on the stream. For GET, the FIN is sent immediately after the request block. For responses, PUT consists of a metadata block containing the uploaded id, expiry, and upload info metadata. GET consists of a metadata block (containing size+expiry+upload info), which is immediately followed by the stream data and FIN. (Thus PUT/GET responses are analogous to the GET/PUT request data in the opposite direction). This initial commit supports read, write, and automatic expiry from disk; the following commits will add support for the non-transfer endpoints (such as updating expiries, querying info, and session version retrieval), which will all happen via the initial (id=0) bt request stream.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a QUIC API allowing for streaming retrieval and delivery of uploads/downloads, for future use with Lokinet.
This server implements file upload and download that uploads and downloads files via QUIC streams, thus allowing both parallel downloads/uploads, as well as allowing more efficient storage as streamed data can be sent piece-by-piece as it loads from disk, rather than needing large all-at-once reads and writes.
This implementation makes heavy use of io_uring for efficiency, and as such will never work outside Linux.
Each upload or download uses exactly one stream (skipping the first stream): an upload is prefixed with a bt-encoded file metadata block (containing the "PUT" request type, size, and optional ttl), while a download request consists of the same sort of block but containing "GET" and the file id.
For PUT, all data on the stream after the initial block is the file data, terminated by a FIN on the stream. For GET, the FIN is sent immediately after the request block.
For responses, PUT consists of a metadata block containing the uploaded id, expiry, and upload info metadata. GET consists of a metadata block (containing size+expiry+upload info), which is immediately followed by the stream data and FIN. (Thus PUT/GET responses are analogous to the GET/PUT request data in the opposite direction).
This initial commit supports upload, download, querying, and extend. The non-file related ancillary endpoints (querying session releases, and token info) are still a work in progress.