Cannot acquire lock: Lock FcntlFlock of /data/ipfs-cluster/cluster.lock failed: resource temporarily unavailable

Help IPFS Cluster
1

This is somewhat related to How to communicate between to clusters in setup

I’m have this docker compose file https://github.com/ipfs/ipfs-cluster/blob/master/docker-compose.yml up and running on a VPS and on my local machine. I’m trying to bootstrap the peer from my VPS to one of the clusters on my local machine, so they can communicate content with each other on a private network. Both my VPS and local machine clusters share the same CLUSTER_SECRET.

On my local machine, when I run

docker exec -it cluster0 ipfs-cluster-service -d daemon --bootstrap /ip4/ip-of-my-VPS/tcp/9096/ipfs/QmS8bGVoARKWtqtF4Wy9Sx9wVDdSMtabZB5XTWEVPdGQX8

or

docker exec -it cluster1 ipfs-cluster-service -d daemon --bootstrap /ip4/ip-of-my-VPS/tcp/9096/ipfs/QmS8bGVoARKWtqtF4Wy9Sx9wVDdSMtabZB5XTWEVPdGQX8

I get the same error.

cannot acquire lock: Lock FcntlFlock of /data/ipfs-cluster/cluster.lock failed: resource temporarily unavailable

error obtaining execution lock: cannot acquire lock: Lock FcntlFlock of /data/ipfs-cluster/cluster.lock failed: >resource temporarily unavailable. If no other ipfs-cluster-service process is running, remove /data/ipfs->cluster/cluster.lock, or make sure that the config folder is writable for the user running ipfs-cluster-service.

locking not initialized, unlock is noop

Are there any suggestions or workarounds to this?

2

It seems you want to execute that command on an onready running container. The containers start ipfs-cluster-service and the error you are seering is precisely not being able to launch another peer because one is already running.

The solution here is to launch your container like docker run ipfs/ipfs-cluster daemon --bootstrap /ip4/ip-of-my-VPS/tcp/9096/ipfs/QmS8bGVoARKWtqtF4Wy9Sx9wVDdSMtabZB5XTWEVPdGQX8 so that the new peer bootstraps directly to the other one at launch.

Note the docker compose launches two cluster peers and two ipfs daemons locally and it’s just a way to quickly get a 2-peer cluster to play with. In your setup I’d use docker run directly or first modify the compose file to run only one peer and one ipfs daemon.

3

@hector Thanks for the prompt response

I was able to bootstrap it after tweaking the compose file, replacing the container name with the ip of my VPS, like so

        if [ ! -d /data/ipfs-cluster/raft ]; then
          while ! ipfs-cluster-ctl --host /dns4/<ip-of-my-vps>/tcp/9094 id; do
            sleep 1
          done
          pid=`ipfs-cluster-ctl --host /dns4/<ip-of-my-vps>/tcp/9094 id | grep -o -E "^(\w+)"`
          sleep 10
          cmd="daemon --bootstrap /dns4/<ip-of-my-vps>/tcp/9096/ipfs/$$pid"
        fi

When I log the peers on my VPS now, I can see the extra peer.

From my local machine, however, I get

QmNQjufbKRiGDtiHLiFcScYQZxEpC6vtj8f6Qxj8UM2jwj | ERROR: dial backoff

and I’m still unable to share content.

This here is the response body from my local Docker ipfs client, not sure if it’s useful.

Response body:[  
   {  
      "id":"QmNQjufbKRiGDtiHLiFcScYQZxEpC6vtj8f6Qxj8UM2jwj",
      "addresses":null,
      "cluster_peers":null,
      "cluster_peers_addresses":null,
      "version":"",
      "commit":"",
      "rpc_protocol_version":"",
      "error":"dial backoff",
      "peername":""
   },
   {  
      "id":"QmZUydUPq5TdfsAkMFwgRjx9LAeE7whegg1i89azTqRChm",
      "addresses":[  
         "/ip4/127.0.0.1/tcp/9096/ipfs/QmZUydUPq5TdfsAkMFwgRjx9LAeE7whegg1i89azTqRChm",
         "/ip4/172.27.0.4/tcp/9096/ipfs/QmZUydUPq5TdfsAkMFwgRjx9LAeE7whegg1i89azTqRChm"
      ],
      "cluster_peers":[  
         "QmNQjufbKRiGDtiHLiFcScYQZxEpC6vtj8f6Qxj8UM2jwj",
         "QmZUydUPq5TdfsAkMFwgRjx9LAeE7whegg1i89azTqRChm",
         "QmfJqKUTGUjMNNKFVwVQPF8SXCr2TqrDxPi51bMiY4wRgp"
      ],
      "cluster_peers_addresses":[  
         "/ip4/127.0.0.1/tcp/9096/ipfs/QmNQjufbKRiGDtiHLiFcScYQZxEpC6vtj8f6Qxj8UM2jwj",
         "/ip4/172.27.0.5/tcp/9096/ipfs/QmNQjufbKRiGDtiHLiFcScYQZxEpC6vtj8f6Qxj8UM2jwj",
         "/ip4/127.0.0.1/tcp/9096/ipfs/QmfJqKUTGUjMNNKFVwVQPF8SXCr2TqrDxPi51bMiY4wRgp",
         "/ip4/172.20.0.5/tcp/9096/ipfs/QmfJqKUTGUjMNNKFVwVQPF8SXCr2TqrDxPi51bMiY4wRgp"
      ],
      "version":"0.10.1+git543656aa832d6399e72f87f20e840d16abe93c11",
      "commit":"",
      "rpc_protocol_version":"/ipfscluster/0.10/rpc",
      "error":"",
      "ipfs":{  
         "id":"Qmb16i3yy3sD4p8BMM8DkcUisdLF9UTFRjZzLShR47RXdK",
         "addresses":[  
            "/ip4/127.0.0.1/tcp/4001/ipfs/Qmb16i3yy3sD4p8BMM8DkcUisdLF9UTFRjZzLShR47RXdK",
            "/ip4/172.27.0.2/tcp/4001/ipfs/Qmb16i3yy3sD4p8BMM8DkcUisdLF9UTFRjZzLShR47RXdK",
            "/ip4/<ip-of-my-vps>/tcp/1085/ipfs/Qmb16i3yy3sD4p8BMM8DkcUisdLF9UTFRjZzLShR47RXdK"
         ],
         "error":""
      },
      "peername":"8fa38cf90383"
   },
   {  
      "id":"QmfJqKUTGUjMNNKFVwVQPF8SXCr2TqrDxPi51bMiY4wRgp",
      "addresses":[  
         "/ip4/127.0.0.1/tcp/9096/ipfs/QmfJqKUTGUjMNNKFVwVQPF8SXCr2TqrDxPi51bMiY4wRgp",
         "/ip4/172.20.0.5/tcp/9096/ipfs/QmfJqKUTGUjMNNKFVwVQPF8SXCr2TqrDxPi51bMiY4wRgp"
      ],
      "cluster_peers":[  
         "QmNQjufbKRiGDtiHLiFcScYQZxEpC6vtj8f6Qxj8UM2jwj",
         "QmZUydUPq5TdfsAkMFwgRjx9LAeE7whegg1i89azTqRChm",
         "QmfJqKUTGUjMNNKFVwVQPF8SXCr2TqrDxPi51bMiY4wRgp"
      ],
      "cluster_peers_addresses":[  
         "/ip4/127.0.0.1/tcp/9096/ipfs/QmNQjufbKRiGDtiHLiFcScYQZxEpC6vtj8f6Qxj8UM2jwj",
         "/ip4/172.27.0.5/tcp/9096/ipfs/QmNQjufbKRiGDtiHLiFcScYQZxEpC6vtj8f6Qxj8UM2jwj",
         "/dns4/<ip-of-my-vps>/tcp/9096/ipfs/QmZUydUPq5TdfsAkMFwgRjx9LAeE7whegg1i89azTqRChm"
      ],
      "version":"0.10.1+git543656aa832d6399e72f87f20e840d16abe93c11",
      "commit":"",
      "rpc_protocol_version":"/ipfscluster/0.10/rpc",
      "error":"",
      "ipfs":{  
         "id":"QmcbgkvYStCNjyJaGW47YJAaw57Qhw7k5EiHQ466EDXDZ5",
         "addresses":[  
            "/ip4/127.0.0.1/tcp/4001/ipfs/QmcbgkvYStCNjyJaGW47YJAaw57Qhw7k5EiHQ466EDXDZ5",
            "/ip4/172.20.0.2/tcp/4001/ipfs/QmcbgkvYStCNjyJaGW47YJAaw57Qhw7k5EiHQ466EDXDZ5"
         ],
         "error":""
      },
      "peername":"c680e18055d7"
   }
]
4

I see three peers in that response when there should be two?. It would seem that the VPC peer is not able to dial back to your local peer?

Might help to run with --net=host, at least to debug. dial-backoff means that the port is closed but if you say that “it was able to bootstrap” it would mean that the connection was stablished so I’m unsure why it would not re-use it when it needs to talk back to that peer. Or maybe you are running that command from the third peer which cannot cut through NAT.

tl;dr; ensure connectivity exists between all peers on :9096 (and don’t run 3 peers on the same place if you just want to run 2).