Go-ds-crdt: A distributed key-value store implementation for IPFS

Ecosystem
,
1

Some weeks ago we finished go-ds-crdt as the new datastore backend for the IPFS Cluster shared state. Since them we’ve been polishing it a bit.

go-ds-crdt provides a distributed key-value store using CRDTs and implements the Datastore interface. The idea is that any datastore can now be replaced by a replicated datastore. go-ds-crdt is agnostic to IPFS/libp2p and just needs a way to broadcast updates to all the replicas and to fetch CID-addressed (IPLD) blocks. Of course, it can work with libp2p PubSub for broadcasting and with IPFS for announcing / retrieving blocks (or IPFS-lite).

Although I have not used OrbitDB personally, it is based on the same principles, except so far we did not have anything like this in Go-land.

8 Likes
2

Hello @hector! Thanks for the very nice work and concise explanation on this. I’ve been currently experimenting with go-ds-crdt and it’s incredibly useful to my use case.

I do have a question though and I think you might be the one to help me with that: when putting blocks to the CRDT datastore, considering I have setup a proper bitswap exchange, would it be possible to fetch these blocks from the IPFS daemon running locally?

From my experiments, I have tried connecting to the daemon from my nodes, as well as the other way around, and whenever I try to do a ipfs block get #cid, it hangs. Am I correct in assuming that Kubo in this case is not handling all types of blocks but instead things that could be built into unifxs nodes? In my case, I have dag-pb blocks.

3

No, Kubo should handle anything ipld, dag-pb included.

There’s a lot to unwrap here. How are you running go-ds-crdt? Are you using it as “datastore” for an ipfs implementation running on top? i.e. the IPFS-lite instance you configure go-ds-crdt with is not concerned with the content you put in the datastore (i.e other blocks), it is only concerned about the Merkle-CRDT blocks.

Theoretically, if you have a working go-ds-crdt datastore, you could use it to be the datastore of an IPFS node too (i.e. another instance of IPFS-lite initialized with the go-ds-crdt datastore). Sharing the same ipfs-lite might be tricky (to initialize ipfs-lite you need a datastore, and to initialize go-ds-crdt you need ipfs-lite). You would have to implement a SessionDAGService wrapping a not-yet-created ipfs-lite instance (which gets set after having the go-ds-crdt). Does this answer your question at all?

1 Like
4

Thanks for the prompt and very thorough response @hector! This indeed answers a lot for me!

In my specific case, I’m actually trying to scaffold some minor things (the least possible stuff needed to interact with the network). Currently I’m using go-ds-crdt as the DB layer and basically just bootstrapping my own libp2p host and DHT (looking at how Kubo builds and configures it, as reference).

I’m then trying to connect to my own ipfs daemon node as a bootstrap node of my libp2p small application. That works fine, and I can see both connections are setup properly and also the bitswap protocol is supported. However, when putting blocks to the crdt store from my app, the blocks are not fetchable from ipfs daemon (ipfs block get $cid hangs).

Maybe I’m doing something wrong or misinterpreting how everything is supposed to be glued together. In terms of the DAGService, I’m using merkledag.DagService and not using ipfs-lite (which has some upgrade issues and limitations too).

The high-level code of setting everything up looks a bit like this:

bs := blockstore.NewBlockstore(sync.MutexWrap(datastore.NewMapDatastore()))
network := bitswapnet.NewFromIpfsHost(host, routing)
exchange := bitswap.New(ctx, network, bs, bitswap.ProvideEnabled(true))
blsrv := blockservice.New(bs, exchange)
dag := merkledag.NewDAGService(blsrv)

And the bitswap stat looks like this after I try accessing the cid from the daemon’
s web UI:

bitswap status
	provides buffer: 0 / 256
	blocks received: 0
	blocks sent: 0
	data received: 0
	data sent: 0
	dup blocks received: 0
	dup data received: 0
	wantlist [1 keys]
		bafybeiaiv6b5bo5hkwehvrm2ppyzkqs6kmkynfyb2w2iqcuugklmmkhali
	partners [2]

Hope that clears the scenario for you to understand it a little bit better. Thanks!

5

If I understand well, you are creating a go-ds-crdt and then putting blocks on it. Whatever you are putting on the go-ds-crdt datastore will not get replicated unless you have another go-ds-crdt datastore peer somewhere else.

But also, what you put in the go-ds-crdt datastore is not something that is announced and remotely fetchable by other ipfs daemons. If you want that you can put your blocks directly through your Blockstore which has a the exchange and everything on top. The go-ds-crdt you have created does not have a blockstore/exchange/blockservice/bitswap on top of it.

go-ds-crdt will make Merkle-CRDT dag nodes available to the ipfs network. These are used to wrap and replicate the things that you are writing to the go-ds-crdt datastore, but those things are opaque and may not well be blocks etc, so they are not interpreted or announced to the network as blocks.

1 Like
6

And as I mentioned, yes you could put blockstore/blockservice/bitswap on top of go-ds-crdt by using go-ds-crdt to create a different, separate ipfs-lite instance (with a separate libp2p host and ID), for example.

2 Likes
7

The below question is moved here: Using blockservice.New with echange parameter to store online

Hi, my question might be noob but this is the only piece of code I found on how to use blockservice.New with echange parameter.
Basically I want to create a blockservice which stores data on IPFS network. Can someone please guide me on how can I achieve it with a sample piece of code?
I want to use the created blocksservice in WNFS and store on IPFS instead of locally