Kingfisher Collect incremental data and log backups

[jpmckinney]

When it looked like ocp23 (data support) wouldn't come back online, I considered whether we had backups for everything.

• Reading https://ocdsdeploy.readthedocs.io/en/latest/deploy/servers/data-support.html#create-a-data-support-main-server I notice a few things we ought to backup with this method: https://ocdsdeploy.readthedocs.io/en/latest/develop/update/backup.html
  • /home/incremental/data/
  • /home/incremental/logs/
• There are also the Kingfisher Collect logs for non-incremental crawls in /home/collect/scrapyd/logs. @yolile, when you have a moment, I am curious to know your opinion on backing those up. We don't currently migrate them across servers, but perhaps recent ones are useful for debugging recent crawls.

The other candidates for backups were:

• Kingfisher Collect downloaded files: These are mostly duplicates of what's loaded in the DB, so okay to lose.
• local-load directory contents: These are also loaded into the DB, so okay to lose.
• RabbitMQ state: I think we would want to restart collection processing or re-run spiders in this scenario.
• Miscellaneous log files (Docker, etc.): I think these are okay to lose.

The items in this second list aren't mentioned in our server migration docs, and have never been migrated in the past.

[yolile]

There are also the Kingfisher Collect logs for non-incremental crawls in /home/collect/scrapyd/logs. @yolile, when you have a moment, I am curious to know your opinion on backing those up. We don't currently migrate them across servers, but perhaps recent ones are useful for debugging recent crawls.

Hmm, when we see issues, we document them here via Sentry anyway. The only other useful aspect of keeping them is comparing error rates and types among crawls, but I guess we can always check the number of items, etc, in the database instead.

[jpmckinney]

Okay, for now we can just back up the incremental crawls. The team member-initiated crawls are somewhat more transient.