State of pubsub

Is there any news regarding pubsub? When will it become a non-experimental feature that can be relied on? Are there any plans to use a better algorithm than floodsub?

At actyx, we are planning to build a system on top of ipfs that is going to need some kind of pubsub mechanism. It would be very nice to be able to use pubsub for this, but if it is not reliable we would have to revert to a centralised alternative like mqtt.

3 Likes

Same situation, we have to adopt a temporary solution waiting for pubsub to be non-experimental and more stable. For dapps and normal apps, this is really nice feature to integrate with IPFS!

Is there a roadmap for features and planned goals?

2 Likes

Out of curiosity: what pubsub sytem are you using? Sadly it thinks that pubsub will not be production ready in time for us to use it.

I would also really much like a roadmap.

Apache Kafka seems to be the best solution at the moment…

Honestly I am not expecting PubSub from IPFS to have the same features as Kafka/RabbitMQ. What I would love is a stabler implementation to remove centralized piece of tech, and ensure that the message arrives at destination :slight_smile:

Same here. Kafka has lots of guarantees that we don’t need or want. We don’t need exactly-once, not even at-least-once. We don’t even need order guarantees. We want the pubsub just to send around hashes, so if a message gets lost it will just “heal” as soon as the next gets through.

Just some reasonable effort to get the messages to the subscribers.

Kafka is much too heavy for us, which is why we are considering MQTT / mosquitto.

Hi! At this point, we feel pretty confident in recommending people use Pubsub—it should be stable today and we plan to try and make upcoming improvements backwards compatible. (If you’re actively experiencing serious issues, please file them at go-floodsub/issues or js-libp2p-floodsub/issues!)

In particular, the next major improvement, Gossipsub (https://github.com/libp2p/go-floodsub/pull/67), will be backwards compatible, so current Floodsub peers will be able to communicate with Gossipsub peers. If you start using the current pubsub implementation today, you should be able to slowly and progressively upgrade all your peers/nodes/devices without needing to coordinate anything special. (Gossipsub is also actually a part of the go-floodsub library, not a separate module or package you’ll have to use. It’ll also still be included and accessible within go-ipfs and js-ipfs.) We hope to release for Go in the next few months, then we’ll start working on the JS version (in the mean time, the backwards-compatibility should keep things working).

Gossipsub aims primarily to make pubsub work better in large networks (roughly 10,000-100,000 nodes) and adds the ability for nodes to relay messages, so nodes with fewer connections can still participate in a topic.

After Gossipsub, we’re turning our attention to optimizing performance when there are a small or fixed number of publishers/sources for messages. This’ll really help use cases like streaming live video or individual sensors that stream their data to the world via pubsub, but doesn’t add anything for cases where there are lots of publishers and subscribers, like a blockchain.

In a more general scope, we want to focus on keeping pubsub low-level and simple, with a focus on performance, decentralization, and scale. That means we don’t plan on doing things like long-lived queues (we are adding small buffers to improve reliability), guaranteed ordering, and so on. If you need those features, you can still build them on top of libp2p/IPFS pubsub, but we don’t plan to directly support them. (If you’re working on a higher-level library that adds these sorts of things, we’d love to hear about it!)

8 Likes

And hopefully it goes without saying, but we certainly plan on keeping pubsub well supported and as good as it can be. It’s not something we think of as experimental now—it’s an important part of Libp2p and IPFS.

1 Like

It would be nice to have more documentation together with releases :smiley:

You know… to actually use these tools :smiley: (I am mostly using go btw)

We are working on it! https://github.com/ipfs/docs/issues/58

1 Like

That’s great news. Maybe we won’t need a centralised broker after all.

Awesome! This is exactly what we need. It is relatively straightforward to build stronger guarantees on top of robust and high performance primitives with fewer guarantees.

Thanks a lot for your work. Despite frequenting this forum regularly and even checking the commit history of go-ipfs occasionally, I was not aware of this.