-
Notifications
You must be signed in to change notification settings - Fork 18
refactor: Replace Arduino HTTP Client with esp native #391
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
hhvrc
wants to merge
37
commits into
develop
Choose a base branch
from
feature/ditch-arduino-httpclient
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
657ab37
Initial commit
hhvrc ff81cf6
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc 22dc664
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc 8ceac76
Use helpers
hhvrc 042f28d
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc 50e317f
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc 9dcc162
Some more stuff
hhvrc 49be717
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc eb734c7
Push WIP
hhvrc c342d31
More work
hhvrc 664a589
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc 9604c39
More work
hhvrc d7774cc
More work on custom http client
hhvrc 0d966e3
More wooooorkkkkk AAAAAAAAAAAAAAAAAAA
hhvrc 979a17d
These are not needed lmao
hhvrc 8cc417b
nvm...
hhvrc c0ee503
more work
hhvrc e8d2790
More fixes and touchup
hhvrc d83d888
Update HTTPClientState.cpp
hhvrc 3a274f5
More work
hhvrc bb36316
Arduino is and always has been a shitty ecosystem
hhvrc c44bbc0
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc 0bfc377
Revert nitpick
hhvrc 454a508
Revert more nitpicking
hhvrc a6bc157
Pass along headers and Retry-After
hhvrc 7e79d41
Fix more issues
hhvrc 614e4c8
Last touchup
hhvrc 68cf530
Update include/http/HTTPClient.h
hhvrc b59e4b9
Update src/util/DomainUtils.cpp
hhvrc 435e6ef
Update src/util/DomainUtils.cpp
hhvrc 3579cee
Update src/serial/command_handlers/authtoken.cpp
hhvrc c9f2094
Update src/util/ParitionUtils.cpp
hhvrc da86842
Fixed url assignment issues
hhvrc 3de2c16
Update platformio.ini
hhvrc f752a17
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc 441688f
Merge branch 'develop' into feature/ditch-arduino-httpclient
hhvrc 6b85a36
Experiement with global CA store
hhvrc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| #pragma once | ||
|
|
||
| #include <cstdint> | ||
| #include <functional> | ||
|
|
||
| namespace OpenShock::HTTP { | ||
| using DownloadCallback = std::function<bool(std::size_t offset, const uint8_t* data, std::size_t len)>; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| #pragma once | ||
|
|
||
| #include "Common.h" | ||
| #include "http/HTTPClientState.h" | ||
| #include "http/HTTPResponse.h" | ||
| #include "http/JsonResponse.h" | ||
| #include "RateLimiter.h" | ||
|
|
||
| #include <esp_err.h> | ||
|
|
||
| #include <cstdint> | ||
| #include <memory> | ||
|
|
||
| namespace OpenShock::HTTP { | ||
| class HTTPClient { | ||
| DISABLE_COPY(HTTPClient); | ||
| DISABLE_MOVE(HTTPClient); | ||
|
|
||
| public: | ||
| HTTPClient(const char* url, uint32_t timeoutMs = 10'000) | ||
| : m_state(std::make_shared<HTTPClientState>(url, timeoutMs)) | ||
| { | ||
| } | ||
|
|
||
| inline esp_err_t SetUrl(const char* url) { | ||
| return m_state->SetUrl(url); | ||
| } | ||
|
|
||
| inline esp_err_t SetHeader(const char* key, const char* value) { | ||
| return m_state->SetHeader(key, value); | ||
| } | ||
|
|
||
| inline HTTPResponse Get() { | ||
| auto response = m_state->StartRequest(HTTP_METHOD_GET, 0); | ||
| if (response.error != HTTPError::None) return HTTP::HTTPResponse(response.error, response.retryAfterSeconds); | ||
|
|
||
| return HTTP::HTTPResponse(m_state, response.statusCode, response.contentLength, std::move(response.headers)); | ||
| } | ||
| template<typename T> | ||
| inline JsonResponse<T> GetJson(JsonParserFn<T> jsonParser) { | ||
| auto response = m_state->StartRequest(HTTP_METHOD_GET, 0); | ||
| if (response.error != HTTPError::None) return HTTP::JsonResponse<T>(response.error, response.retryAfterSeconds); | ||
|
|
||
| return HTTP::JsonResponse(m_state, jsonParser, response.statusCode, response.contentLength, std::move(response.headers)); | ||
| } | ||
|
|
||
| inline esp_err_t Close() { | ||
| return m_state->Close(); | ||
| } | ||
| private: | ||
| std::shared_ptr<HTTPClientState> m_state; | ||
| }; | ||
| } // namespace OpenShock::HTTP | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| #pragma once | ||
|
|
||
| #include "Common.h" | ||
| #include "http/DownloadCallback.h" | ||
| #include "http/HTTPError.h" | ||
| #include "http/JsonParserFn.h" | ||
| #include "http/ReadResult.h" | ||
|
|
||
| #include <cJSON.h> | ||
|
|
||
| #include <esp_http_client.h> | ||
|
|
||
| #include <map> | ||
| #include <string> | ||
| #include <string_view> | ||
|
|
||
| namespace OpenShock::HTTP { | ||
| class HTTPClientState { | ||
| DISABLE_COPY(HTTPClientState); | ||
| DISABLE_MOVE(HTTPClientState); | ||
| public: | ||
| HTTPClientState(const char* url, uint32_t timeoutMs); | ||
| ~HTTPClientState(); | ||
|
|
||
| esp_err_t SetUrl(const char* url); | ||
|
|
||
| esp_err_t SetHeader(const char* key, const char* value); | ||
|
|
||
| struct HeaderEntry { | ||
| std::string key; | ||
| std::string value; | ||
| }; | ||
|
|
||
| struct [[nodiscard]] StartRequestResult { | ||
| HTTPError error{}; | ||
| uint32_t retryAfterSeconds{}; | ||
| uint16_t statusCode{}; | ||
| bool isChunked{}; | ||
| uint32_t contentLength{}; | ||
| std::map<std::string, std::string> headers{}; | ||
| }; | ||
|
|
||
| StartRequestResult StartRequest(esp_http_client_method_t method, int writeLen); | ||
|
|
||
| // High-throughput streaming logic | ||
| ReadResult<uint32_t> ReadStreamImpl(DownloadCallback cb); | ||
|
|
||
| ReadResult<std::string> ReadStringImpl(uint32_t reserve); | ||
|
|
||
| template<typename T> | ||
| inline ReadResult<T> ReadJsonImpl(uint32_t reserve, JsonParserFn<T> jsonParser) | ||
| { | ||
| auto response = ReadStringImpl(reserve); | ||
| if (response.error != HTTPError::None) { | ||
| return response.error; | ||
| } | ||
|
|
||
| cJSON* json = cJSON_ParseWithLength(response.data.c_str(), response.data.length()); | ||
| if (json == nullptr) { | ||
| return HTTPError::ParseFailed; | ||
| } | ||
|
|
||
| T data; | ||
| if (!jsonParser(json, data)) { | ||
| return HTTPError::ParseFailed; | ||
| } | ||
|
|
||
| cJSON_Delete(json); | ||
|
|
||
| return data; | ||
| } | ||
|
|
||
| inline esp_err_t Close() { | ||
| if (m_handle == nullptr) return ESP_FAIL; | ||
| return esp_http_client_close(m_handle); | ||
| } | ||
| private: | ||
| static esp_err_t EventHandler(esp_http_client_event_t* evt); | ||
| esp_err_t EventHeaderHandler(std::string key, std::string value); | ||
|
|
||
| esp_http_client_handle_t m_handle; | ||
| bool m_reading; | ||
| uint32_t m_retryAfterSeconds; | ||
| std::map<std::string, std::string> m_headers; | ||
| }; | ||
| } // namespace OpenShock::HTTP |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| #pragma once | ||
|
|
||
| namespace OpenShock::HTTP { | ||
| enum class HTTPError { | ||
| None, | ||
| ClientBusy, | ||
| InternalError, | ||
| RateLimited, | ||
| InvalidUrl, | ||
| InvalidHttpMethod, | ||
| NetworkError, | ||
| ConnectionClosed, | ||
| SizeLimitExceeded, | ||
| Aborted, | ||
| ParseFailed | ||
| }; | ||
|
|
||
| inline const char* HTTPErrorToString(HTTPError error) { | ||
| switch (error) | ||
| { | ||
| case HTTPError::None: | ||
| return "None"; | ||
| case HTTPError::ClientBusy: | ||
| return "ClientBusy"; | ||
| case HTTPError::InternalError: | ||
| return "InternalError"; | ||
| case HTTPError::RateLimited: | ||
| return "RateLimited"; | ||
| case HTTPError::InvalidUrl: | ||
| return "InvalidUrl"; | ||
| case HTTPError::InvalidHttpMethod: | ||
| return "InvalidHttpMethod"; | ||
| case HTTPError::NetworkError: | ||
| return "NetworkError"; | ||
| case HTTPError::ConnectionClosed: | ||
| return "ConnectionClosed"; | ||
| case HTTPError::SizeLimitExceeded: | ||
| return "SizeLimitExceeded"; | ||
| case HTTPError::Aborted: | ||
| return "Aborted"; | ||
| case HTTPError::ParseFailed: | ||
| return "ParseFailed"; | ||
| default: | ||
| return "Unknown"; | ||
| } | ||
| } | ||
| } // namespace OpenShock::HTTP |
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| #pragma once | ||
|
|
||
| #include "Common.h" | ||
| #include "http/DownloadCallback.h" | ||
| #include "http/HTTPClientState.h" | ||
| #include "http/JsonParserFn.h" | ||
| #include "http/ReadResult.h" | ||
|
|
||
| #include <cstdint> | ||
| #include <map> | ||
| #include <memory> | ||
| #include <string> | ||
|
|
||
| namespace OpenShock::HTTP { | ||
| class HTTPClient; | ||
| class [[nodiscard]] HTTPResponse { | ||
| DISABLE_DEFAULT(HTTPResponse); | ||
| DISABLE_COPY(HTTPResponse); | ||
| DISABLE_MOVE(HTTPResponse); | ||
|
|
||
| friend class HTTPClient; | ||
|
|
||
| HTTPResponse(std::shared_ptr<HTTPClientState> state, uint16_t statusCode, uint32_t contentLength, std::map<std::string, std::string> headers) | ||
| : m_state(state) | ||
| , m_error(HTTPError::None) | ||
| , m_retryAfterSeconds(0) | ||
| , m_statusCode(statusCode) | ||
| , m_contentLength(contentLength) | ||
| , m_headers(std::move(headers)) | ||
| { | ||
| } | ||
| public: | ||
| HTTPResponse(HTTPError error) | ||
| : m_state() | ||
| , m_error(error) | ||
| , m_retryAfterSeconds() | ||
| , m_statusCode(0) | ||
| , m_contentLength(0) | ||
| , m_headers() | ||
| { | ||
| } | ||
| HTTPResponse(HTTPError error, uint32_t retryAfterSeconds) | ||
| : m_state() | ||
| , m_error(error) | ||
| , m_retryAfterSeconds(retryAfterSeconds) | ||
| , m_statusCode(0) | ||
| , m_contentLength(0) | ||
| , m_headers() | ||
| { | ||
| } | ||
|
|
||
| inline bool Ok() const { return m_error == HTTPError::None && !m_state.expired(); } | ||
| inline HTTPError Error() const { return m_error; } | ||
| inline uint32_t RetryAfterSeconds() const { return m_retryAfterSeconds; } | ||
| inline uint16_t StatusCode() const { return m_statusCode; } | ||
| inline uint32_t ContentLength() const { return m_contentLength; } | ||
|
|
||
| inline ReadResult<uint32_t> ReadStream(DownloadCallback downloadCallback) { | ||
| auto locked = m_state.lock(); | ||
| if (locked == nullptr) return HTTPError::ConnectionClosed; | ||
|
|
||
| return locked->ReadStreamImpl(downloadCallback); | ||
| } | ||
|
|
||
| inline ReadResult<std::string> ReadString() { | ||
| auto locked = m_state.lock(); | ||
| if (locked == nullptr) return HTTPError::ConnectionClosed; | ||
|
|
||
| return locked->ReadStringImpl(m_contentLength); | ||
| } | ||
|
|
||
| template<typename T> | ||
| inline ReadResult<T> ReadJson(JsonParserFn<T> jsonParser) | ||
| { | ||
| auto locked = m_state.lock(); | ||
| if (locked == nullptr) return HTTPError::ConnectionClosed; | ||
|
|
||
| return locked->ReadJsonImpl(m_contentLength, jsonParser); | ||
| } | ||
| private: | ||
| std::weak_ptr<HTTPClientState> m_state; | ||
| HTTPError m_error; | ||
| uint32_t m_retryAfterSeconds; | ||
| uint16_t m_statusCode; | ||
| uint32_t m_contentLength; | ||
| std::map<std::string, std::string> m_headers; | ||
| }; | ||
| } // namespace OpenShock::HTTP |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new HTTPClient implementation does not integrate with the rate limiting system. The old implementation called
_getRateLimiter()and checkedrateLimiter->tryRequest()before making requests. Without this integration, the application may exceed API rate limits and get blocked by the server. Consider integrating the RateLimiters::GetRateLimiter functionality into the HTTPClient.