`routing::gossip::tests::test_channel_timeouts` is flaky

[TheBlueMatt]

On Windows CI (ie a really slow machine), we get

failures:

---- routing::gossip::tests::test_channel_timeouts stdout ----
thread 'routing::gossip::tests::test_channel_timeouts' panicked at 'assertion failed: network_graph.removed_channels.lock().unwrap().is_empty()', lightning\src\routing\gossip.rs:2550:13


failures:
    routing::gossip::tests::test_channel_timeouts

test result: FAILED. 582 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out; finished in 32.11s

[valentinewallace]

Also running into this on #1812, but in benchmark: https://github.com/lightningdevkit/rust-lightning/actions/runs/3732711432/jobs/6332523097