Skip to content

Handle the case where a process is SIGKILLed #85

@phargogh

Description

@phargogh

I have a case with the NCI work where my taskgraph-based pipeline is exhausting the available memory and linux's oom-killer is effectively kill -9ing the process. When this happens:

  1. python starts up a new process to replace it, keeping the number of Processes the same as what the Pool expects at all times. The new process started has a different PID.
  2. Because the Task was killed, the self.task_done_executing event is never set, which means that the graph hangs indefinitely.

The behaviorally consistent thing to do here might just be to terminate the graph when we detect that a process has been killed.

I'm labeling this as an enhancement because it's new behavior and because the workaround here is to just not exhaust the memory I'm allowed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions