Roadmap for asynchronous BEDbase/BEDboss data processing

[nsheff]

We need these components for asynchronous processing of BED files to populate bedbase.

1. Automatic registering of new BED files posted to GEO.

• we are already automatically creating a PEP with new BED files using github action
• bbuploader process reads from PEPhub (run by bedboss geo upload-all)
• implement --light CLI arg for bedboss geo upload --light ... (compute ID, upload BED file, collect and input metadata)
• add new github action that will run this process weekly.
• make sure current endpoints properly ignore "stub" records, if needed (those that haven't yet gone through full/heavy process).

2. Allowing users to register new BEDsets

• endpoint for registering a BEDset
  • POST a PEPhub registry path. BEDbase retrieves this PEP, validates against BEDbase::bedset schema, and then creates a new BEDset in the database. (This relies completely on pephub auth, so no further auth is required?) This should 1. validate against PEP schema. Do other validations. Limit size for now to, say 2k BEDs? maybe limit throughput? Maybe to start, hard code a list of "allowed" pephub namespaces.
  • Create a button on BEDbase cart page to "Create BEDset PEP for this Cart". (This would require user to authenticate with PEPhub...)

3. Daemon that retrieves unprocessed BED files and processes them (plots and statistics)
   • endpoint for unprocessed files ?
   • endpoint for unprocessed plots
   • endpoint for unprocessed statistics
   • script thats hits these endpoints, and then does the correct processing (in BEDboss)
   • wrapper daemon to sleep and wrap the above script.
4. Daemon that retrieves unprocessed BED sets and processes them.
   • endpoint for retrieving which BEDsets are registered, but not processed
   • script thats hits bedset endpoints, and then does the correct processing (in BEDboss)
   • wrapper daemon to sleep and wrap the above script.

[khoroshevskyi]

To run unprocessed bed files that are in the database I implemented the function in bedboss:

bedboss reprocess-all ...

[khoroshevskyi]

Fixes needed before closing this issue:

• Release new geniml
• Fix Switch bedToBigBed to bigTools written in rust bedboss#89
• Fix warning for Python 3.12 -> Release 0.14.4 pypiper#232
• Start cron job for the docker container
• Fix duplicated logs databio/bedboss@05d54d5
• Add logging what number of files currently have been processed databio/bedboss@3dff8ca
• Pipier should start once, not for each file (in reprocessing). databio/bedboss@4b4fe1b
• Do not run command (bigbed) if genome is not avaliable. databio/bedboss@5e9a308
• Fix duplication in logging databio/bedboss@05d54d5
• Make R script as separate process
• Check file for content before processing it

[khoroshevskyi]

This issue is fixed.
Lite processing of the GEO bed files is happening on Github actions. Heavy processing is running on AWS as cron job in docker.