Retrieve all data that is stored on a specific CID without having to make a GET req to each path

Help IPFS
1

Hi,

I’ve recently discovered IPFS and was looking into how one would be able to recover / obtain a master meta data file from a CID which has many paths e.g. https://ipfs.io/ipfs/{CID}/1 - https://ipfs.io/ipfs/{CID}/1000. I first tried this with normal HTTP GET requests and realised how slow it was, then tried it with asynchronous requests and came across the problem of getting rate limited very quickly. So now was wondering if there would be a better approach to this through IPFS itself or if I would need my own IPFS node to tackle this problem. If someone could give me some insight into this I would appreciate it.

1 Like
2

With a gateway IDK I’m not super familiar.

You could build indexing with IPLD or use a directory to wrap around all the files.

With a directory it would look like this ipfs cat <CID>/folder/filename.ext.

With IPLD metadata it can be anything you want but you’ll have to built it yourself.
ipfs dag get <CID>/array/35/map/key_one/data

4

There is no way to do this with gateways, but you can use ipfs get <CID>, potentially archiving the output with the --archive flag.

You can also use GitHub - ipfs/ipget: Retrieve files over IPFS and save them locally..