-
Notifications
You must be signed in to change notification settings - Fork 129
update storage extension to v2.0.0 #1561
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
update storage extension to v2.0.0 #1561
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1561 +/- ##
==========================================
- Coverage 92.22% 92.21% -0.02%
==========================================
Files 55 55
Lines 8414 8538 +124
Branches 973 986 +13
==========================================
+ Hits 7760 7873 +113
- Misses 466 476 +10
- Partials 188 189 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
gadomski
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we have unnecessary new extensions here ... both schemes and refs can just be classes, not extensions themselves?
| class StorageSchemeType(StringEnum): | ||
| AWS_S3 = "aws-s3" | ||
| CUSTOM_S3 = "custom-s3" | ||
| AZURE = "ms-azure" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you also add GCS = "gcp-gs"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
generally, having parity to the v1 Enum would be great
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't see best practice GCP storage defined here, so did not add as a possible enum here, but as written schemes can be set to any string. I think opening a PR in the extension w/ a suggested schema to use for GCP storage would make sense before implemented here.
1b0a9ab to
a0267df
Compare
gadomski
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe I missed it, but we should test the migration path from v1 to v2.
docs/api/extensions.rst
Outdated
| storage.StorageSchemesExtension | ||
| storage.StorageRefsExtension |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like the docs need to be reverted back to a single class?
pystac/extensions/storage.py
Outdated
| Returns the dictionary encoding of this object | ||
| Returns: | ||
| dict[str, Any |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing closing brace.
I'm taking a look at what adding a migration hook would require, but I think there will be some cases that will be harder to migrate cleanly. (e.g. azure where we need account id, or the platforms that exist in v1 but not v2 e.g. IBM, alibaba) |
I think raising a warning in those cases and then leaving the data unchanged will be sufficient. |
Description:
Updates storage extension to v2.0.0. Due to the major changes in v2, I have not attempted to make migrations from v1 to v2
StorageScheme platform property allows templated uri's, so arbitrary additional properties are allowed to be set on StorageScheme objects
PR Checklist:
pre-commit run --all-files)pytest)