I am trying to set up a cluster so that I can duplicate data stored on my IPFS nodes in case one fails.
I have set up a basic cluster that currently only has 2 nodes. I built the cluster on top of an existing IPFS node that has quite a few pins (3000+, ~5GB of data). I initialized the cluster, ensured that the nodes were able to communicate, and then proceeded to call
ipfs-cluster-ctl pin add <CID> for every CID in
ipfs pin ls --type=recursive on
This part seems to have worked. Both
Node 1 and
Node 2 show that they have pinned everything, however when I go directly to my 2nd ipfs node it says the pin does not exist, and the data doesn’t seem to have been transferred as I am only using ~3GB of storage on my 2nd server.
$ ipfs-cluster-ctl pin ls QmegbCAbDCU4eCYYxAzaVZ1qH9KouYEYzHokC7s7TexMJD
QmegbCAbDCU4eCYYxAzaVZ1qH9KouYEYzHokC7s7TexMJD | | PIN | Repl. Factor: -1 | Allocations: [everywhere] | Recursive
$ ipfs pin ls QmegbCAbDCU4eCYYxAzaVZ1qH9KouYEYzHokC7s7TexMJD
Error: path 'QmegbCAbDCU4eCYYxAzaVZ1qH9KouYEYzHokC7s7TexMJD' is not pinned
What am I missing?
Whats the output of?
ipfs-cluster-ctl peers ls
Even if your cluster nodes are sharing the same secret, if you haven’t initiated a bootstrap process on starting node2 by doing
ipfs-cluster-service daemon --bootstrap <cluster-multiaddr-node-1> they will be unaware of each other and subsequently not be part of a cluster.
edit: What ipfs-cluster version are you running?
It definitely is able to find it’s peers. I have bootstrapped
Node 2 and have seen Raft working.
$ ipfs-cluster-ctl peers ls
12D3KooWEsTBprKQcSpe5CoTk7dzmzd2kE1MToetKQEczJDjhCuu | ip-172-31-36-250 | Sees 1 other peers
> IPFS: QmWfrmp4FeY3hoLr1GvpAhjyVdKBwcBEAptVx4QEXviqgp
The two nodes were able to share the pins on the cluster level, but it didn’t seem to trickle down to the underlying IPFS node on
I am running
Ok so you have a valid cluster setup that’s good. What’s the output from
go-ipfs version are you using
I have go-ipfs
v0.4.10. I can upgrade to
v0.4.18 if that would help.
I just restarted everything this morning and
ipfs-cluster-ctl status gives me about 3000 of these:
> ip-172-31-36-250 : PINNED | 2019-06-07T14:18:58.669604866Z
> ip-172-31-33-10 : PIN_QUEUED | 2019-06-07T14:18:57.810174065Z
ipfs pin ls QmZg2b6JVxcZQcfAd3gXuXrNqoqwCPRMShsJDcFFPDKbev still shows it as unpinned.
humm… why are you using such and old version of IPFS?
I would recommend to upgrade ipfs to the latest version (0.4.21).
Pin_queued means your ipfs daemon is pinning something else and this item is waiting. The default configuration is set to fetch+pin at most 10 things in parallel (
concurrent_pins). Everything else goes to
queued and will transition to
ipfs-cluster-ctl status --filter pinning should show you which items are in pinning state. (and
--filter pinned would show which items have finished pinning. If you run from the new ipfs peer
ipfs-cluster-ctl status --local --filter pinned should show everything that is pinned on the new peer).
If connectivity is good between your ipfs peers, there should be progress (
ipfs stats bw shoud show the actual upload/download speeds). Maybe the progress is slower than you expect or simply not working at all, but you’re also using a super old ipfs version so maybe that has something to do with it.
I am running an old version because it was originally set up on top of a pre-existing node. I was hesitant to upgrade because it is being used in production, and I didn’t want any unexpected behavior.
That is helpful. Looks like you are correct. It is pinning now, just very slowly (5-20kb/sec). This would be useful information for the Quickstart tutorial so that people can see it working, or show these options in the
Upgrading will probably help you greatly. We are currently reworking all the docs and will try to mention these things (@pkafei ^^^)
I often get pin_queued. So, what is number limit on the concurrent_pins in the service.json?
On a huge cluster network like 50 nodes are continously pinning, what is the good number for concurrent_pins ?
concurrent_pins controls how many pin requests are triggered against the underlying IPFS daemon at the same time. By default it is 10. If you see pin_queued it would mean that pinning is ongoing for other 10 items in that peer (they should be
pinning). A good number will depend on your usage and how much pressure you want to pass on to the IPFS daemon.
Can we have any priority for pinning ? If it goes into pin_queued state, check for other pins priority and pin this first rather than the rest ?
Is something like that possible ?
That is not possible right now. You can only increase concurrent pins. If you want, you can open a feature request about it.