IPFS nodes in Cluster are not duplicating pinned data/sharing pins

Help IPFS Cluster
1

I am trying to set up a cluster so that I can duplicate data stored on my IPFS nodes in case one fails.

I have set up a basic cluster that currently only has 2 nodes. I built the cluster on top of an existing IPFS node that has quite a few pins (3000+, ~5GB of data). I initialized the cluster, ensured that the nodes were able to communicate, and then proceeded to call ipfs-cluster-ctl pin add <CID> for every CID in ipfs pin ls --type=recursive on Node 1.

This part seems to have worked. Both Node 1 and Node 2 show that they have pinned everything, however when I go directly to my 2nd ipfs node it says the pin does not exist, and the data doesn’t seem to have been transferred as I am only using ~3GB of storage on my 2nd server.

Example:
On Node 2

$ ipfs-cluster-ctl pin ls QmegbCAbDCU4eCYYxAzaVZ1qH9KouYEYzHokC7s7TexMJD
QmegbCAbDCU4eCYYxAzaVZ1qH9KouYEYzHokC7s7TexMJD |  | PIN | Repl. Factor: -1 | Allocations: [everywhere] | Recursive

$ ipfs pin ls QmegbCAbDCU4eCYYxAzaVZ1qH9KouYEYzHokC7s7TexMJD
Error: path 'QmegbCAbDCU4eCYYxAzaVZ1qH9KouYEYzHokC7s7TexMJD' is not pinned

What am I missing?

2

Whats the output of?

ipfs-cluster-ctl peers ls

Even if your cluster nodes are sharing the same secret, if you haven’t initiated a bootstrap process on starting node2 by doing ipfs-cluster-service daemon --bootstrap <cluster-multiaddr-node-1> they will be unaware of each other and subsequently not be part of a cluster.

edit: What ipfs-cluster version are you running?

3

It definitely is able to find it’s peers. I have bootstrapped Node 2 and have seen Raft working.

$ ipfs-cluster-ctl peers ls
12D3KooWEsTBprKQcSpe5CoTk7dzmzd2kE1MToetKQEczJDjhCuu | ip-172-31-36-250 | Sees 1 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/ipfs/12D3KooWEsTBprKQcSpe5CoTk7dzmzd2kE1MToetKQEczJDjhCuu
    - /ip4/172.31.36.250/tcp/9096/ipfs/12D3KooWEsTBprKQcSpe5CoTk7dzmzd2kE1MToetKQEczJDjhCuu
  > IPFS: QmWfrmp4FeY3hoLr1GvpAhjyVdKBwcBEAptVx4QEXviqgp
    - /ip4/127.0.0.1/tcp/4001/ipfs/QmWfrmp4FeY3hoLr1GvpAhjyVdKBwcBEAptVx4QEXviqgp
    - /ip4/127.0.0.1/tcp/8081/ws/ipfs/QmWfrmp4FeY3hoLr1GvpAhjyVdKBwcBEAptVx4QEXviqgp
    - /ip4/18.222.255.9/tcp/4001/ipfs/QmWfrmp4FeY3hoLr1GvpAhjyVdKBwcBEAptVx4QEXviqgp
    - /ip6/::1/tcp/4001/ipfs/QmWfrmp4FeY3hoLr1GvpAhjyVdKBwcBEAptVx4QEXviqgp

The two nodes were able to share the pins on the cluster level, but it didn’t seem to trickle down to the underlying IPFS node on Node 2.

I am running v0.10.1

4

Ok so you have a valid cluster setup that’s good. What’s the output from ipfs-cluster-ctl status
Additionally what go-ipfs version are you using

5

I have go-ipfs v0.4.10. I can upgrade to v0.4.18 if that would help.

I just restarted everything this morning and ipfs-cluster-ctl status gives me about 3000 of these:

QmZg2b6JVxcZQcfAd3gXuXrNqoqwCPRMShsJDcFFPDKbev :
    > ip-172-31-36-250 : PINNED | 2019-06-07T14:18:58.669604866Z
    > ip-172-31-33-10 : PIN_QUEUED | 2019-06-07T14:18:57.810174065Z

However ipfs pin ls QmZg2b6JVxcZQcfAd3gXuXrNqoqwCPRMShsJDcFFPDKbev still shows it as unpinned.

6

humm… why are you using such and old version of IPFS?

I would recommend to upgrade ipfs to the latest version (0.4.21).

Pin_queued means your ipfs daemon is pinning something else and this item is waiting. The default configuration is set to fetch+pin at most 10 things in parallel (concurrent_pins). Everything else goes to queued and will transition to pinning. ipfs-cluster-ctl status --filter pinning should show you which items are in pinning state. (and --filter pinned would show which items have finished pinning. If you run from the new ipfs peer ipfs-cluster-ctl status --local --filter pinned should show everything that is pinned on the new peer).

If connectivity is good between your ipfs peers, there should be progress (ipfs stats bw shoud show the actual upload/download speeds). Maybe the progress is slower than you expect or simply not working at all, but you’re also using a super old ipfs version so maybe that has something to do with it.

7

Thanks hector.

I am running an old version because it was originally set up on top of a pre-existing node. I was hesitant to upgrade because it is being used in production, and I didn’t want any unexpected behavior.

That is helpful. Looks like you are correct. It is pinning now, just very slowly (5-20kb/sec). This would be useful information for the Quickstart tutorial so that people can see it working, or show these options in the ipfs-cluster-ctl docs.

8

Upgrading will probably help you greatly. We are currently reworking all the docs and will try to mention these things (@pkafei ^^^)

9

I often get pin_queued. So, what is number limit on the concurrent_pins in the service.json?
On a huge cluster network like 50 nodes are continously pinning, what is the good number for concurrent_pins ?

Thank you

10

concurrent_pins controls how many pin requests are triggered against the underlying IPFS daemon at the same time. By default it is 10. If you see pin_queued it would mean that pinning is ongoing for other 10 items in that peer (they should be pinning). A good number will depend on your usage and how much pressure you want to pass on to the IPFS daemon.

11

Can we have any priority for pinning ? If it goes into pin_queued state, check for other pins priority and pin this first rather than the rest ?
Is something like that possible ?

Thank you

12

That is not possible right now. You can only increase concurrent pins. If you want, you can open a feature request about it.