Overriding signal handler for SIGINT breaks program without any way to understand in `dask-jobqueue==0.9.0`

[vepadulano]

Describe the issue:

I have a dask-based application, currently too complicated to give a small reproducer, but I can work on that if necessary. A few key steps happening in my program:

• I create one dask.distributed LocalCluster
• I create one dask Client and I attach it to the LocalCluster
• I have a conditional import dask_jobqueue in my import statements. I don't use dask_jobqueue in this particular instance of my program, but it's possible to use it optionally by the user given the right configuration parameter and if the package is installed in the Python environment.
• I create N distinct dask computation graphs
• I submit the N graphs concurrently via a Python thread pool, something akin to

    with concurrent.futures.ThreadPoolExecutor(max_workers=len(ngraphs)) as executor:
        futures = [executor.submit(execute_graph, rootnode) for rootnode in rootnodes]
        concurrent.futures.wait(futures)

I hadn't been using this program for a while, came back to it recently and I found out that it breaks. After many hours spent debugging, I could trace the breaking of my program to whether the dask_jobqueue package is installed in the Python environment or not.

Then I went ahead with git bisect and I found out that this was the offending commit that breaks my program 3a00196

I then noticed there is an unconditional override of the signal handler for SIGINT. This has landed in version 0.9.0, i.e. the latest released version.

I then noticed that the main branch has a different commit, introduced by #668 . This has not landed yet in any officially released version.

I installed a modified version of dask_jobqueue 0.9.0 which just introduces the changes of #668, that is now the signal handler is only overridden in the main thread.

This fixes my program, so I would like to ask to at least publish a version of dask_jobqueue which is not broken.

Anything else we need to know?:

On a more general note, I have serious doubts about the validity of this piece of code. Even with the guard that introduces it only in the main thread, in general there cannot be multiple signal handlers for the same signal. So dask_jobqueue is silently taking over responsibility for managing signal handlers for the entire duration of the program. Has this been specified in the documentation? Furthermore, I can easily imagine many downstream use cases that may want to have their own way to handle SIGINT, which leads to two possible cases:

1. The call to signal.signal done by the downstream library happens after the call done by dask_jobqueue: in this case, the downstream user is possibly happy, but the assumption made by dask_jobqueue will be invalidated, so maybe it will be a problem for this library
2. The last call to signal.signal is done by dask_jobqueue: in this case, the user is unhappy because their assumptions are broken without any good way to understand what happened.

Could you please clarify your opinion on this matter, and could you foresee finding a more secure and isolated way to get to your desired result, without possibly breaking user code silently?

Environment:

• Dask version: 2025.12.0
• Python version: 3.13
• Operating System: Fedora Linux
• Install method (conda, pip, source): pip, source

[jacobtomlinson]

I would like to ask to at least publish a version of dask_jobqueue which is not broken.

I agree, we are overdue a release here. It probably wont get done until the new year though.

So dask_jobqueue is silently taking over responsibility for managing signal handlers for the entire duration of the program.

When being used as a runner then yes this is expected behavior. Happy to update the documentation to make this clearer. I expect it would be on this page. Do you have thoughts on what we should add there?

Furthermore, I can easily imagine many downstream use cases that may want to have their own way to handle SIGINT

I agree when it comes to the cluster managers, but not the runners. I would expect the runners to take full control of process execution and lifecycle.

If you're seeing these problems when using the cluster managers then that's a bug. This should be fixed by #668 but perhaps there are more changes needed.

[vepadulano]

If you're seeing these problems when using the cluster managers then that's a bug.

At this point I'm leaning towards bug rather than lacking documentation. This is a simplified example of what I'm doing (unfortunately not enough yet to really reproduce the issue):

from dask.distributed import Client, LocalCluster, progress
import dask
import concurrent.futures

import dask_jobqueue

def execute_graph(value):
    def mapper():
        return value

    def reducer(a, b):
        return a + b

    dmapper = dask.delayed(mapper)
    dreducer = dask.delayed(reducer)

    mergeables_lists = [dmapper() for _ in range(10)]
    while len(mergeables_lists) > 1:
        mergeables_lists.append(dreducer(mergeables_lists.pop(0), mergeables_lists.pop(0)))

    final_results = mergeables_lists.pop().persist()
    progress(final_results)

    return final_results.compute()


if __name__ == "__main__":

    with Client(LocalCluster(n_workers=16, processes=True, threads_per_worker=1)) as client:
        with concurrent.futures.ThreadPoolExecutor(max_workers=64) as executor:
            futures = [executor.submit(execute_graph, val) for val in range(1, 64)]
            concurrent.futures.wait(futures)
        print(f"Got results: {[f.result() for f in futures]}")

Just removing the import dask_jobqueue in the real application makes it work. Also, patching dask_jobqueue 0.9.0 with #668 makes it work.

I feel like #668 is a good start, but I would consider moving the signal.signal call directly within the BaseRunner class, for example in its __init__.py. As things are right now, just import dask_jobqueue triggers that call, whereas I guess from this statement

I would expect the runners to take full control of process execution and lifecycle.

it derives that this should only happen if the user really wants to use a runner.

[jacobtomlinson]

Yup I agree this is a bug, we shouldn't be setting this handler on import. I've opened #697 to resolve this.

[vepadulano]

Thank you so much for your prompt replies and help 🚀

[jacobtomlinson]

I've merged the fix. I'll make a note to tag a new release first week of January when folks are back from holidays and we can react to any bugs or other fallout from the release.