We are on a 3 node setup, in 1st node ipfs is down. Now we try to add a file then we see pin_error in that particular node and pinned status on the other 2 nodes.
Now we started the ipfs on the 1st node, and did a ipfs-cluster-ctl recover, the status of the hash stuck in pinning state.
How can we recover from this state?
It sounds like recover did what it had to do and is now pinning on IPFS. If pinning is not finishing in a reasonable time, you’d need to check the ipfs daemon on that node directly. Is it correctly connected to the network? Is there something preventing it from finding and downloading the content?
case1 : When we restart the ipfs on 1st node, swarm peers are not getting connected and the hash is in pinning state forever. This happens if we add a file after stoping the ipfs in 1st node.
case 2: If we simply stop the ipfs on 1st node and restart it without adding any file, swarm peers are in sync.
Note: case1 will be resolved if we restart the cluster on 1st node(swarm peers comes up, file also pinned)
This is because Cluster manually triggers a swarm connect operation a few seconds after starting. But the main problem is that your node1 is not discovering the other peers by itself (is this a private network?). You need to make sure node1-ipfs is reachable, that the bootstrapper list is correct and potentially also set the other peers in the configuration (https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#peeringpeers). If they are on the same LAN, you can enable mDNS autodiscovery as long as multicast is supported in that LAN.
As a side note, I don’t see how adding a file would make a difference between case 1 and case 2. The fact that a file was added to the cluster should not affect whether a node connects on boot.