Fastest way to get CID directory

Help
1

Hello, I’m looking for the fastest way to pin a CID directory (that is not mine). So basically I give my app a CID and it downloads the entire directory (usually has 5k-10k files) as fast as possible. The size of the directory is usually only around 5-10mb but for some reason it takes 30 to 60 seconds when I run ipfs pin add <CID>!

Any kind of suggestion is welcome.

2

This depends a lot on the nodes you are downloading from.
For best speed run badgerds ipfs config profile apply badgerds.
And accelerated DHT client (however this should only impact the first discovery time, once you start downloading this should already be over).

1 Like
3

Thanks a lot for answering, I ran badgerds but it’s still taking forever. I noticed that the ipfs.io/ipfs/ gateway most of the time has the files I’m looking for, is there a way to directly connect to this node? Keep in mind I don’t even have to pin the directory locally, only need to fetch it. I store the content of the files in a separate database by making http requests for each file in the directory.

4

is there a way to directly connect to this node?

Yes but pls don’t do that.
If everyone does that ipfs.io will get overloaded even more and slow down to a crawl.

If you really want do to do it (or do it with multiple nodes of yours) checkout: https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#peering

5

Your comment about, “If everyone does that ipfs.io will get overloaded even more and slow down to a crawl.” made me think. Gateways are a nice bride between IPFS and HTTP but is it a bridge too far? It might be nice if there was some sort of coordination between IPFS gateways to share the load like maybe a pub/sub channel that you could announce your gateway and then you could do a 303 redirect to other gateways if it became overloaded or another one had a lower latency.

I guess what I’m thinking is basically to put Public Gateway Checker | IPFS on IPFS and make it automatic and automated. Maybe it already is, I don’t know. Anyway, just a random thought before I’ve had enough coffee to know if it’s a good one.