Fastest way to pull a directory from IPFS

Help
1

Hi,

I am currently working on a project that requires me to download a directory from IPFS. The directories are usually filled with around 10.000 files. Each file is usually not even 1kb in size. The speeds for pinning and then getting a directory are decent (usually around a minute), but I was wondering if there are ways to speed this up. I am aiming to find a way to get everything in under 30 sec. I know IPFS works on a peer-to-peer basis and I thus rely on the speed of other nodes which I can’t control. But are there certain ways to improve the speed at which I can grab data from the directories? I also tried to use the IPFS HTTP gateway distributed over a few servers without any success. Any tips on increasing the pin/get speed are highly appreciated.

1 Like
2

Pinning is the fastest path to grab things as using the same bitswap session for the whole thing optimizes fetching, and dag traversal happens in parallel when possible I think.

If anything, try tuning the configuration (disable sync, and check the bitswap internals options).

1 Like