-
Notifications
You must be signed in to change notification settings - Fork 4
Open
Labels
enhancementNew feature or feature requestNew feature or feature request
Description
Since ProcessQueue spawns a bunch of processes on the local machine, it should be possible to set it up to dispatch the command lines to an AWS batch queue.
Proposed enhancement
Spawn the actual processing codes via a batch queue instead of directly on the machine running ProcessQueue.
IPC required:
- Filesystem needs to be available on the spawning instance and the batch-executing instance
- Full command line goes into the batch queue
- stdout/stderr is ordinarily captured by the spawning ProcessQueue and so this needs to be captured via some other means
- Final created file is normally stored in a temp directory, either ProcessQueue can handle putting this on a shared filesystem or there needs to be some way of moving it across
Alternatives
Could go for fuller clustering support, but this seems like the smallest change that supports good horizontal scaling.
Closure condition
Implemented and tested. Unit testing might be "interesting", either "mock" AWS or pull out the actual batch injection as a very thin shim that is not tested in unit tests (only functional testing).
Metadata
Metadata
Assignees
Labels
enhancementNew feature or feature requestNew feature or feature request