I-ALiRT - Archive cadence #1038
Conversation
| all_items.extend(inst_items) | ||
|
|
||
| dataset = create_xarray_from_records(response["Items"]) | ||
| dataset = create_xarray_from_records(all_items) |
There was a problem hiding this comment.
Does this handle an empty dataset? If we don't get any ialirt packets for a day, would this still create all the expected fields with empty fill values, or do we need to exit out earlier above this so we don't write an empty file?
There was a problem hiding this comment.
Yes. Addressed that empty dataset now.
| INSTRUMENTS = ["mag", "codice_lo", "codice_hi", "hit", "swe", "swapi", "spacecraft"] | ||
|
|
||
|
|
||
| def query_instrument( |
There was a problem hiding this comment.
Don't we already have this in the main API-route? I feel like we should be using the same query function in both places so we don't get out of sync.
There was a problem hiding this comment.
ialirt_ingest.py queries S3 (packet filenames).
ialirt_data_query_api.py queries DynamoDB, but does not paginate.
This archive lambda needs to pull all items in a time window (1 day), so pagination is required.
laspsandoval
force-pushed
the
archive_cadence
branch
from
f45e5d2 to
8d2e629
Compare
2 participants
Change Summary
Overview
Pass in new database, handle pagination, update time to be exactly at midnight
Updated Files
Testing