-
Notifications
You must be signed in to change notification settings - Fork 8
Open
Labels
enhancementNew feature or requestNew feature or request
Milestone
Description
I have a case with the NCI work where my taskgraph-based pipeline is exhausting the available memory and linux's oom-killer is effectively kill -9ing the process. When this happens:
- python starts up a new process to replace it, keeping the number of
Processes the same as what thePoolexpects at all times. The new process started has a different PID. - Because the
Taskwas killed, theself.task_done_executingevent is never set, which means that the graph hangs indefinitely.
The behaviorally consistent thing to do here might just be to terminate the graph when we detect that a process has been killed.
I'm labeling this as an enhancement because it's new behavior and because the workaround here is to just not exhaust the memory I'm allowed.
dcdenu4
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request