Skip to content

Make ProcessQueue AWS-batch aware #131

@jtniehof

Description

@jtniehof

Since ProcessQueue spawns a bunch of processes on the local machine, it should be possible to set it up to dispatch the command lines to an AWS batch queue.

Proposed enhancement

Spawn the actual processing codes via a batch queue instead of directly on the machine running ProcessQueue.

IPC required:

  • Filesystem needs to be available on the spawning instance and the batch-executing instance
  • Full command line goes into the batch queue
  • stdout/stderr is ordinarily captured by the spawning ProcessQueue and so this needs to be captured via some other means
  • Final created file is normally stored in a temp directory, either ProcessQueue can handle putting this on a shared filesystem or there needs to be some way of moving it across

Alternatives

Could go for fuller clustering support, but this seems like the smallest change that supports good horizontal scaling.

Closure condition

Implemented and tested. Unit testing might be "interesting", either "mock" AWS or pull out the actual batch injection as a very thin shim that is not tested in unit tests (only functional testing).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or feature request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions