Collection cache to avoid checking the same links many times #86
Description
One of the slow operations is checking if the collection contains a specific task already. This requires communication with Redis and is done for every task we find. We may be able to improve the performance by given each worker a local cache with links that they have already confirmed are in the collection.