Loading a batch of data from IPFS (Dag)

Help IPFS
1

Hi :slight_smile:

We created a social media app on top of IPFS that is similar to Medium. We store posts as JSON objects to DAG, and we save images as files (not DAG) to IPFS. We load images through IPFS gateway and it works fast and images get cached by the browser. But posts work not so good. We cannot load them via IPFS gateway (because a post is stored as DAG?). And looks like the posts are not cached by the browser. Can you help us with the next questions, please?

  1. Does the content caches when itā€™s loaded from IPFS via the ā€œGETā€ request? AFAIK, but still not sure, only images that we fetch directly from IPFS Gateway are cached.
  2. If only content from IPFS Gateway is cached, then how to access it using Dag? Iā€™m asking because we cannot fetch content from the gateway using CID returned by Dag.
  3. Is there a way to load a batch of data from IPFS providing, for example, an array of CIDs?

You can track the issue in the network tab (DevTools) on the main page, or this one if you set pagination to 100 elements/page.

Hereā€™s how it looks like:

Screenshot from 2020-10-06 18-23-16
Screenshot from 2020-10-06 18-23-16666Ɨ628 95.4 KB

1 Like
2

Hello IPFS team! We need your help :pleading_face:

1 Like
3

Itā€™s best to use ipfs.dag.get or ipfs.dag.tree to pull the contents of a dag.

Check the Documentation

Weā€™re developing a comprehensive social network framework on IPFS @ questnetwork.github.io/docs - Itā€™s still going to take us a few weeks until we have all the features from Discord, Facebook, Twitter, etc but in 0.9.4 weā€™re already using dags for the timeline. In the future, you can always check the example app and library for how to do most operations on IPFS from the browser. You might even want to use our engine in your app once weā€™re production ready, itā€™s open source and we publish every release to NPM.
Check out if it could work for you, maybe you want to join us, work together on the library and use our library for your app too =)

4

Check the headers attached to the request. What do they say? I would expect they are the same as for images, and they indicate that content is immutable and therefore cacheable. By IPFS Gateway do you mean fetching content via :8080 (gateway port) vs :5001 (API port) ? I donā€™t understand why you would not be able to load a post (or any CID-addressed content) from the gateway.

Maybe there is difference in headers as exposed by the gateway and by the API, and maybe thatā€™s the issue with caching.

Why not? What is the problem?

No, but these days connections have keep-alives and are re-used so thatā€™s not a huge source of delays. What it seems from the screenshot is that your application triggers requests sequentially, and perhaps it could trigger them in parallel or in batches?

5

Why not? Some gateways expose a read-only sub-set of the API on their gateway port. I donā€™t know if the dag APIs are included in that subset or not. Nor do I know if/where might be a definitive list of which APIs are exposed. But itā€™s easy enough to test with curl if you have a dag CID handy!

6

Just tested and these all work (note especially the last one):

curl -X POST http://ipfs.io/api/v0/block/get?arg=QmNdvaN7JxwjUL6eKmG5ugeq5F7ikwvKgagXDubSzFBSw4

curl -X POST http://ipfs.io/api/v0/block/stat?arg=QmNdvaN7JxwjUL6eKmG5ugeq5F7ikwvKgagXDubSzFBSw4

curl -X POST http://ipfs.io/api/v0/cat?arg=QmNdvaN7JxwjUL6eKmG5ugeq5F7ikwvKgagXDubSzFBSw4

curl -X POST http://ipfs.io/api/v0/dag/get?arg=QmNdvaN7JxwjUL6eKmG5ugeq5F7ikwvKgagXDubSzFBSw4

7

By Gateway I mean exactly :8080/ipfs.
For example [working link] and [link with Dag provided].

8

Thatā€™s actually not what I meant:

  • You used the default IPFS CID in these examples.
  • You use IPFS API, not IPFS Gateway.

Check this message

9

We make requests in parallel using Promise.all(), but it looks like IPFS node still responds sequentially to our parallel requests.

1 Like
10

Hello friends, any suggestions?

11

Does your app not have a ā€˜backendā€™? The normal way this would be done is that your backend piece would collect all the data it needs to send to the browser all at once, in one request to load the entire page.

12

@F3Joule - I donā€™t use gateways a lot, so donā€™t know any tricks. But if spinning up an IPFS node in the browser is an option: Have a look at our Coral module in https://questnetwork.github.io/docs -> Coral Module -> Open in GitHub, sometimes weā€™re iterating over thousands of dag nodes from the browser and never had issues.

13

Yes, we have a backend. But this is centralization. We thought that we can get high performance by directly requesting data from IPFS. But this DAG seems working very slow.

14

We need to get content from a remote IPFS node. And content is stored in DAG. We see that our UI loads DAG objects sequential and donā€™t know why, because we use Promise.all(). What are we doing wrong?

15

By GitHub code, do you mean this file?

16

Yep, more importantly the getObject method and this part:

let  encryptedHex = await this.ipfsNode.dag.get(hash);

We have a timeout in our method but you can also call the ipfs.dag.get method with a timeout parameter

1 Like
17

@StationedInTheField btw, where can we chat about your networking app?

18

In general, imo, trying to submit 100s or 1000s of calls to a server at the same time is not a desirable pattern, and is one of the reasons I never fell in love with RESTful[ness] either. I realize images are processed this way and pages can have lots of images, but still the thought of populating some arbitrary GUI list in an app by doing one HTTP call per item just sounds like thatā€™s never going to be very fast.

But if youā€™re doing something where you cannot even create a back end method to call, to get all the data at once, you may just be stuck with slowness. Also remember, even if you can get your network to submit 100s of requests simultaneously, that still doesnā€™t even guarantee a faster performance than submitting them all sequentially. It all depends on the capabilities of the server side of whatever youā€™re calling, and lots of servers will just serialize requests coming over the same session away, for threading reasons.

19

BTW: Hereā€™s a great way to cache information in the browser, thatā€™s a wrapper around IndexedDB, as a key/value store:

20

@smn on the network but I also just created a gitter for it, you can also ask your questions and share your suggestions there: https://gitter.im/QuestNetwork/qOS

1 Like