2 years struggling with IPFS... And how I succeed using it with ScuttleButt!

First time I tried IPFS on my computer was by using “ipfs companion”. As a beginner, I uploaded a few files in it, and they appear in my browsing list. Everything was going right…

On a second computer, in the same LAN, I installed “go-ipfs”.
I used ipfs get CLI commands to retrieve my previously published files. And it works!! I was thinking, great!

But I realize that it was missing indexes.

  • On 1st computer, every data was using MFS (ipfs files ls)
  • On 2nd computer this structure is not maintained anymore…

This was very confusing… But I didn’t care then.

To benefit from “decentralized storage” I was willing to “pin” my data on multiple nodes.
So I made experiments.

First, I tried “ipfs cluster”. It works pretty well… But as my node storage wasn’t equal on each, the smallest was limiting the whole. This configuration is good if you have nodes in different regions of earth and maintain data consistency from everyplace. It was not my personal need. I didn’t want to buy as much as same size disks to replicate data on every nodes!!

Then I tried “textile” as cafe was promising to make a more precise pin management applied. This time, data replication was better… But, protocol was hard to understand and a new problem occurs. If my swarm nodes didn’t share the same ~/.ipfs/swarm.key I get performance troubles.

It was about DHT leaking… When my network reach a certain point, ipfs swarm peers gets inconsistent between nodes… Nodes I didn’t know about was entering my swarm. IPNS publish was slow. In fact, nothing really works. So sharing the same swarm.key was the only mean I found to control bootstrap list and swarm peers. I used this configuration for a while. But adding a new node, sending this key and maintaining security process was a pain!!

At the end, my IPFS swarm was working, but impossible to easily extend.

But last January, I discovered ScuttleBot and realize it was implementing network control I was missing in IPFS. Adding “User Accounts” (as textile), but with an easier to use protocol and far more library available!!

ScuttleButt protocol is like a light ipfs, with more options, but “blob” storage limitation up to 8MB. So I decided to mix them.

With some simple bash tasks, every node is getting its SSB “friends list” and apply it to ipfs bootstrap and ipfs swarm peers. It was excellent!! Now I could remove my swarm.key as no more DHT leaking was happening… SO GREAT!

As I was working with Libre Money and didn’t want to wait for Filecoin… I start making some “nano chains” into ipfs (to memorize accountings) in a mutable filesystem I called zenTAG. So I was able to create an economy for my files :wink:

I am now working on a protocol fat stack over Debian to normalize it. I am welcoming anyone interested in joining my experiments.


Now adding a new Node in my IPFS swarm is as simple as becoming SSB friends :wink:

  1. Check if node is following Pub (or bootstrap)
sbot links --dest @PUBPublicKeyBase64encoded=.ed25519 --source @PublicKeyBase64EncodedOfMyNewNode=.ed25519 --rel contact --values
  1. Follow Pub
sbot publish --type contact --contact @PUBPublicKeyBase64encoded=.ed25519 --following

And reverse from “Pub/Bootstrap” Node

Now IPFS swarm DHT is kept synchronized between all my nodes automatically :wink:

But adding a file to IPFS needs to be related to an “index”.
For example, in case of music, tThis could be a json based on “mp3tags” (artists, album, song name, …).
But it could be nicer to have an “audio” index based on audio fingerprinting

Dejavu would allow IPFS data to be linked and found like Shazam !!!

Anyone would like to make it with me?

I’m sure the www.Audius.co team will be interested.

@Akita it would be a good idea to join effort. Do you have a contact?

No I don’t. You will have to go through the contact list on their website, I guess?

Here is a blog post about them, too:

This is why ipfs cannot be interplanetary and fast dynamic.
Using scuttlebutt protocol for network partitioning provide the freedom to choose which of the properties are prefered :wink: