Question about content distribution in IPFS

Ecosystem
1

Hi all,
I spent my whole yesterday to go deep dive into IPFS source code. Now I can understand most part of it. However, I still have a question about its protocol and hope someone can help me clear about this.
As I understand, when I want to put a block of data into IPFS, this is what will happen:

  1. I save block in my store.
  2. I create a message AddProvider to notify that I’m a provider of this block. This message will be sent to top closest peers to the key (CID of the block).
  3. Peer who received this message will save in provider store which says that I’m a provider of this block.
    So this is the way information of a block will broadcast to the whole network. However, I am not clear about how content will be distributed in IPFS. DHT seems only provide way to find providers of a given block. Bitswap protocol only controls which block I can provide and which blocks I want, so it doesn’t specify how my block will be distributed in IPFS.
    So currently my understand is: until someone get that block and pin that block, I’m the only one will provide that block. This is sad because I want my content will be automatically distributed in IPFS.
    Thanks for your help.
3 Likes
2

Content is not automatically distributed because that would break the autonomy of my node (my node downloads only what I tell it to download). This is important to prevent accidentally hosting illegal content. To pin content over multiple nodes you can run a cluster, ask nicely, or pay for it.

3 Likes
3

I want to implement something on top of IPFS library, so I want to stick with IPFS module rather than IPFS cluster. Right now, there is only whitepaper about Filecoin and no concrete software, so it’s hard to know more about it, e.g when it’s available.
In their paper, they said about Bitswap strategy. It states that:

The protocol must also incentivize nodes to seed when
they do not need anything in particular, as they might have
the blocks others want. Thus, BitSwap nodes send blocks to
their peers optimistically, expecting the debt to be repaid.

This means that you can broadcast your content to others and help store their content in return.
It’s fair.
So according to your explanation, right now, we might face with scalability problem when many one want your content at same time ?

4

It might be slow at first if you publish new content and expect half the world to download it right away. But if the content is popular, it will spread through the caches of other nodes and some might even pin it. So blocks in high demand tend to grow in supply (a nice property).

Just like torrents, you can’t force or depend on any other node to seed your content. Bitswap incentivizes nodes to seed blocks they already have, but they don’t go randomly downloading new blocks to seed.

5

Hi @piedar,
In our case, latency is critical (latency since some one put content and the others can get it). An at a time, many people might want to request for it. So I might want to modify Bitswap protocol a little bit. Any suggestion without any modification?
Thanks.

6

From my (limited) understanding, latency is not yet strong point of IPFS. Really I think you’ll have to prototype your use case and see if the performance is acceptable. You might consider setting up a fast semi-public gateway to pin your users’ content and get it out there faster.

7

This was never implemented. We may do this in the future but it has major legal implications and may simply be unworkable.

So, there are two issues to latency:

  1. Finding the data.
  2. Fetching the data.

Finding the data is currently very slow. We use the providers system as you say but we need to optimize it.

As for the second issue, we really need to work on this but it’s non-trivial.

There are a bunch of things we can do without modifying the protocol. This includes:

  • Asking specific peers for blocks instead of all peers.
  • Randomize which blocks to fetch first (bittorrent does this as it helps spread out requests as early as possible).

You’ll find issues like this in the {go,js}-ipfs repos.

We’re also looking to improve the protocol by allowing nodes to ask for sets of blocks (described by a short query) instead of individual blocks. This should help when, e.g., traversing a large path (reduce the number or round trips). @vmx is the go-to person on this subject.

3 Likes
8

Thank you for the information about Bitswap protocol.

9

If you are looking for a storage protocol similar to IPFS that has monetary incentivization, look up SWARM on the Ethereum blockchain. When V0.4 is released it will have full integration of its Swap Swear And Swindle account protocol.