When researching the ipfs ecosystem, it appears to me that anyone who wants to add new functionality to nodes needs to fork the default implementation, write their own version or convince Protocol Labs to add it to the default implementation.
As a result, I’ve seen lots of different node implementations where the non-default parts essentially become inoperable with each other (unless you try and meld together multiple customisations). This feels like a missed opportunity for the ecosystem as a whole, assuming my observation above is correct.
Whilst I’m sure there are a number of use cases where forking or writing a custom implementation are the correct way to go, I’m left with the impression that many interesting things being done (whether that be the Peergos encryption layer, the Crust network’s incentive layer etc) are siloed from each other. My expectation was that it would be possible to extend the default node implementation (E.g. through a plugin architecture) to maximise innovation.
This popped up a number of questions in my head:
What elements of the default node implementations are extensible?
Is it currently possible to add functionality that can be easily re-used by default node users?
If no to the above, is there a reason a pluggable architecture has not been embraced?
I think all of this is because we failed to explain what IPFS is and what matters.
IPFS is not go-ipfs, and your goal shouldn’t be to merge back with PL’s implementation.
For information go-ipfs does 0 ipfs related protocol, everything is handled by libs like go-bitswap, go-libp2p, … go-ipfs is mainly A LOT of glue code to provide APIs and configs.
Everything, EXCEPT, that mainly at build time and making that available at runtime like a few parts already are is really complicated and slow, I don’t think that worth it.
Yes if you count importing new packages and setuping some inits as easy (which it is if you already know how to code, it’s mainly just reading docs and connecting plumbing together).
It exists, but this is not go-ipfs. (checkout ipfs-lite or a DIY solution instead)
Discovering more use cases than originally intended is not bad per se. In the case of go-ipfs because it includes and exposes as APIs many additional things like pubsub for example it serves more as backend as a service and platform as a service. We need something like this as a foundation for connecting all interoperable things in the ecosystem. For example I forked it in order to be able to do an orbit-db integration and expose it via the GitHub - ipfs/go-ipfs-api: The go interface to ipfs's HTTP API shell.
Having said that go-ipfs architecture is 10 years old and it’s showing. As a glue layer it’s an ideal candidate for being rewritten from a single binary CLI app into a local cluster where you can plug in micro services and fine tune resource usage per use case. I think we need to account for the difference between plugins and micro services. Plugins are dependencies, micro services are independent instances.
A plugin is usually developed either by the protocol creator or by a participant in the ecosystem that wants to be interoperable. Going down the plugins path is about centralization and makes IPFS a registry for plugins. Making go-ipfs a local cluster keeps things decoupled so that everyone can integrate anything themselves.
IPFS-lite looks great at what it does. It’s KISS at it’s best but it doesn’t seem to aim at solving the lack of interoperability in the ecosystem.
@Jorropo appreciate your explanation. The modularity of the protocol layers is one of my favourite things about ipfs, so I do see that provides a lot of flexibility.
If I may, I’d just like to take one example to understand it a bit better. One of the most often talked about subjects I’ve seen is a content encryption/privacy layer on top of ipfs. Would it be fair to say that it’s possible to create a standalone module/package (language agnostic, since I’m focusing on the js version more initially, but would also make use of go) that could be imported to extend any existing node implementation that already does what you highlighted and imports the underlying protocol modules?
Assuming the answer there is yes, then I wonder if there is something to your original point of guiding people in how to create (and make use of) reusable, extensible functionality? As far as I’m aware, that isn’t covered explicitly in the documentation?
I also think that might extend to “discovery of” community-written modules (E.g. ability to easily find “encryption” implementations, which also encourages the creation of standalone modules). I noticed in the ipfs docs there is a section talking about why encryption isn’t a core protocol layer, as well as links to some projects that have implemented encryption themselves, but nothing that points to packages or modules created for that singular purpose. It would seem beneficial if there was somewhere on ipfs.io where people could search for and submit modules, to help with that.