How to communicate between to clusters

Hi guys,

I did set up ipfs-clusters on two computers. One is on my local machine and the other is on VPS. What I try to undestand and achieve is how to replicate files from my local machine to VPS and vice versa. However, it seems like I couldn’t do it and kindly need your help. Here are what I have done

  1. ipfs and ipfs-cluster are installed on both machines. They all run in daemon with no issues. I also update secret key in order to help 2 computers can communicate each other.
  2. I did update service.json file in ipfs-cluster folder of both machines
    a. my VPS

Blockquote
“init_peerset”: [
“QmS17ZTNx1FJisJnmcuuVYRyoWyAJbxs1X2yC39hdtkz33” (this is my local machine)
]

b. my local machine

Blockquote
“init_peerset”: [
“QmS17ZTNx1FJisJnmcuuVYRyoWyAJbxs1X2yC39hdtkz35” (this is my VPS)
]

Then I tried to add file on my local and wait whether this added file will be pinned into VPS machine. But it didn’t work. I am sure that something wrong in my configuration file.

Thank you so much for your help in advance

Regards,

Hello, have you set up the peerstore file so that both peers know how to contact each other? (https://cluster.ipfs.io/documentation/starting/#requirements).

Can you check that once started they’re both aware and connected? (ipfs-cluster-ctl peers ls should list full info for both). At least one of your peers should be reachable for the other (open tcp/9096 in your vps)

Once cluster is started, you need to ipfs-cluster-ctl pin add <cid> your content in order to be pinned in both places (after being added to ipfs), or you need to add it through the cluster proxy (explained here: https://cluster.ipfs.io/documentation/composite-clusters/#the-ipfs-proxy).

If all that doesn’t work, I will need full logs, but usually following the documentation should get you there.

Also, it may help to run ipfs-cluster-service state clean before retrying, otherwise it may not initialize the init_peerset in raft correctly. The log messages on startup should say if it’s initializing or not.

Dear Hector,

Thank you so much for your help. I am much appreciated ^^

Let me be more specific my case

  1. on my server machine:
    I run IPFS and ipfs-cluster-service daemon
  2. on my local machine:
    I run ipfs-cluster-service daemon --bootstrap /ip4/my-server-ip/tcp/9096/ipfs/my-server-cluster-id

Then I see the log file

  • INFO service: Bootstrapping to /ip4/my-server-ip/tcp/9096/ipfs/my-server-cluster-id
  • INFO consensus: peer is ready to join a cluster consensus
    ERROR cluster: ***** ipfs-cluster consensus start timed out (tips below)
    This might be due to one or several causes:
    • Check the logs above this message for errors
    • Check that there is connectivity to the “peers” multiaddresses
    • Check that all cluster peers are using the same “secret”
    • Check that this peer is reachable on its “listen_multiaddress” by all peers
    • Check that the current cluster is healthy (has a leader). Otherwise make
      sure to start enough peers so that a leader election can happen.
    • Check that the peer(s) you are trying to connect to is running the
      same version of IPFS-cluster.

Just confirm that

  • Check that all cluster peers are using the same “secret” (they are the same)
  • Be able to connect to server with port 9096
  • It should be 3 nodes for testing cluster?

Best regards,
Quang

Hello,

There should be more error lines printed in your logs, with specific libp2p connection errors. Can you run the peers with --loglevel debug and send the full startup logs for both peers? Which OS are you using?

(while you may want to obfuscate IPs, there is no need to obfuscate your Peer IDs).

Dear Hector

The full error logs are as follows on my local machine:

10:11:01.516  INFO    service: Initializing. For verbose output run with "-l debug". Please wait... app.go:485
10:11:01.516 DEBUG    service: checking lock daemon.go:58
10:11:01.516 DEBUG    service: Success! ipfs-cluster-service lock acquired daemon.go:58
10:11:01.517 DEBUG     config: raft section configuration loaded config.go:304
10:11:01.517 DEBUG     config: restapi section configuration loaded config.go:305
10:11:01.517 DEBUG     config: ipfshttp section configuration loaded config.go:306
10:11:01.517 DEBUG     config: maptracker section configuration loaded config.go:308
10:11:01.517 DEBUG     config: monbasic section configuration loaded config.go:309
10:11:01.517 DEBUG     config: pubsubmon section configuration loaded config.go:309
10:11:01.517 DEBUG     config: disk section configuration loaded config.go:311
10:11:01.517 DEBUG     config: numpin section configuration loaded config.go:311
10:11:01.518  INFO  consensus: cleaning empty Raft data folder (/Users/duyquang/.ipfs-cluster/raft) state.go:169
10:11:01.523 DEBUG  consensus: starting Consensus and waiting for a leader... daemon.go:120
10:11:01.523 DEBUG  consensus: creating libp2p Raft transport raft.go:92
10:11:01.523 DEBUG  consensus: creating BoltDB store raft.go:97
10:11:01.524 DEBUG  consensus: creating raft snapshot store raft.go:97
10:11:01.525 DEBUG  consensus: creating Raft consensus.go:70
10:11:01.526  INFO    cluster: IPFS Cluster v0.4.0 listening on:
        /ip4/127.0.0.1/tcp/9096/ipfs/QmVH6r1wwwh2aGjZsteX7dHdt8Bxk6M3SSbAhiH4SUUVxf
        /ip4/10.1.125.33/tcp/9096/ipfs/QmVH6r1wwwh2aGjZsteX7dHdt8Bxk6M3SSbAhiH4SUUVxf
        /ip4/169.254.246.31/tcp/9096/ipfs/QmVH6r1wwwh2aGjZsteX7dHdt8Bxk6M3SSbAhiH4SUUVxf

 daemon.go:134
10:11:01.526 DEBUG  consensus: checking for existing raft states consensus.go:152
10:11:01.526  INFO    restapi: REST API (HTTP): /ip4/127.0.0.1/tcp/9094 asm_amd64.s:2361
10:11:01.526  INFO   ipfshttp: IPFS Proxy: /ip4/127.0.0.1/tcp/9095 -> /ip4/127.0.0.1/tcp/5001 asm_amd64.s:2361
10:11:01.526  INFO    service: Bootstrapping to /ip4/104.131.92.107/tcp/9096/ipfs/QmczvDg6SEbRjQ8DSeGmCew714yhfdNG7SDsZz2755ycmY asm_amd64.s:2361
10:11:01.526 DEBUG  consensus: staging servers do not need initialization consensus.go:152
10:11:01.526 DEBUG    cluster: Join(/ip4/104.131.92.107/tcp/9096/ipfs/QmczvDg6SEbRjQ8DSeGmCew714yhfdNG7SDsZz2755ycmY) daemon.go:153
10:11:01.526  INFO  consensus: peer is ready to join a cluster consensus.go:152
10:11:01.526  INFO    restapi: REST API (libp2p-http): ENABLED. Listening on:
        /ip4/127.0.0.1/tcp/9096/ipfs/QmVH6r1wwwh2aGjZsteX7dHdt8Bxk6M3SSbAhiH4SUUVxf
        /ip4/10.1.125.33/tcp/9096/ipfs/QmVH6r1wwwh2aGjZsteX7dHdt8Bxk6M3SSbAhiH4SUUVxf
        /ip4/169.254.246.31/tcp/9096/ipfs/QmVH6r1wwwh2aGjZsteX7dHdt8Bxk6M3SSbAhiH4SUUVxf

 asm_amd64.s:2361
10:11:21.526 ERROR    cluster: ***** ipfs-cluster consensus start timed out (tips below) ***** cluster.go:133
10:11:21.527 ERROR    cluster: 
**************************************************
This peer was not able to become part of the cluster.
This might be due to one or several causes:
  - Check the logs above this message for errors
  - Check that there is connectivity to the "peers" multiaddresses
  - Check that all cluster peers are using the same "secret"
  - Check that this peer is reachable on its "listen_multiaddress" by all peers
  - Check that the current cluster is healthy (has a leader). Otherwise make
    sure to start enough peers so that a leader election can happen.
  - Check that the peer(s) you are trying to connect to is running the
    same version of IPFS-cluster.
**************************************************
 cluster.go:133
10:11:21.527  INFO    cluster: shutting down Cluster cluster.go:369
10:11:21.527  INFO  consensus: stopping Consensus component cluster.go:440
10:11:21.527 DEBUG  consensus: Raft state is catching up to the latest known version. Please wait... raft.go:411
10:11:21.527 DEBUG  consensus: current Raft index: 0/0 raft.go:411
10:11:21.527 ERROR       raft: NOTICE: Some RAFT log messages repeat and will only be logged once logging.go:71
10:11:21.527 ERROR       raft: Failed to take snapshot: nothing new to snapshot logging.go:48
10:11:21.528  INFO    monitor: stopping Monitor cluster.go:461
10:11:21.528  INFO    restapi: stopping Cluster API cluster.go:466
10:11:21.528  INFO   ipfshttp: stopping IPFS Proxy cluster.go:470
10:11:21.528  INFO pintracker: stopping MapPinTracker cluster.go:475
10:11:21.528 DEBUG    monitor: publishing metric ping to pubsub. Expires: 1531847511528460853 cluster.go:233
10:11:22.517 DEBUG    service: Successfully released execution lock daemon.go:93

I am using OS Mac Sierra. IPFS-Cluster version 0.4.0.

Note: I did re-initialize ipfs-cluster on both machines. Just start from scratch again

It would seem that 104.131.92.107:9096 is not reachable from your local machine. Your cluster peer does not seem to manage to open a connection to it (I guess the other peer’s logs show nothing?).

I think, if you increase wait_for_leader_timeout long enough for the network requests to timeout, you will see the specific error (timeout). The bootstrap process estimates 20 seconds is long enough and therefore fails before any specific log messages show in the logs.

1 Like

Thanks Hector,

Thank you so much. I did forget the basic one. They can talk together now

Regards,
Quang

1 Like