Fetching data quite slow, even with direct connections

[whyrusleeping]

Over in filecoin land we are using ipget to download snark parameters. These files are (currently) around 1.5 GB, and take quite a long time (often in the 20 minute range) using the following invocation:

ipget --node=temp --progress -p=/ip4/138.201.67.219/tcp/4002/ws/ipfs/QmUd6zHcbkbcs7SMxwLs48qZVX3vpcM8errYS7xEczwRMA -p=/ip4/138.201.67.218/tcp/4002/ws/ipfs/QmbVWZQhCGrS7DhgLqWbgvdmKN7JueKCREVanfnVpgyq8x -p=/ip4/94.130.135.167/tcp/4002/ws/ipfs/QmUEMvxS2e7iDrereVYc5SWPauXPyNwxcy9BXZrC1QTcHE -p=/ip4/138.201.68.74/tcp/4001/ipfs/QmdnXwLrC8p1ueiq2Qya8joNvk3TVVDAut7PrikmZwubtR -p=/ip4/138.201.67.220/tcp/4001/ipfs/QmNSYxZAiJHeLdkBg38roksAR9So7Y5eojks1yjEcUtZ7i QmNSuxq15JPFCTehxVpgJydNZ79rpLoNwnLzQMGA9EziXg

Realistically, directly connecting to the peers with the data and pulling it from them on my connection should take ~30 seconds, i'm fine with a small amount of overhead, so if it took a minute it would be fine.

These files are going to get much bigger in the nearish future, even up to 100GB, and people will be required to download them. We need this process to be as fast as possible.

[jbenet]

Important note: this is a big source of pain for filecoin devs and users at the moment. not really clear what is holding the connection up -- is it bitswap perf?

[olizilla]

This may be related to: ipfs/go-bitswap#187

we're not parallelizing reads from disk. Given sufficient send buffers we'll be blocked on disk reads, not network writes.

[olizilla]

For context: the nodes that the command above connects to (/ip4/138.201.67.219 et al) are our ipfs-cluster nodes. Due to the issue above, it may that they are currently unable to send data as fast as the network would allow.