I have successfully setup one IPFS cluster + IPFS node in a VPS, and another IPFS cluster + IPFS node is a separate VPS machine. Both are setup with the provided docker-compose.yml example, with the same secret and cdtr.
The second was started with command: daemon --bootstrap /ip4/2nd-VPS-IP/tcp/exposed-port/p2p/peer-id. The peers connected successfully but only the peerstore of the second peer was populated (with the multiaddress provided as bootstrap). The peerstore of the first peer remained empty. After restarting each peer alternatively to test reconnections the peerstore of the first peer was populated with:
Why isn’t the first peer aware of the IP and port of the second peer?
Is the peerstore meant to be populated with the information of all peers in the cluster?
Also, when running an IPFS cluster dockerized with its 9096 port exposed, it never shows a multiaddress with its external IP; when an IPFS typically does.
The second was started with command: daemon --bootstrap /ip4/2nd-VPS-IP/tcp/exposed-port/p2p/peer-id
Shouldn’t it be 1st-VPS-IP and not 2nd ?
The peerstore is only saved on shutdown I think. From your description and the fact that the public IP does not show among the multiaddresses etc, I am inclined to say that both peers did not get connected?
Did ipfs-cluster-ctl peers ls show both peers? Do they show the right addresses there?
Both peers successfully connected, were displayed on each other’s ipfs-cluster-ctl peers ls and replicated state.
The peerstore is only saved on shutdown I think
Thanks for the suggestion, I will investigate if the ipfs cluster process is being shut down correctly.
Also, why isn’t there an endpoint in the IPFS cluster API to dynamically add peers? Given that this feature is in IPFS https://docs.ipfs.io/reference/api/http/#api-v0-bootstrap-add, is there a reason why is not / can’t be implemented in the IPFS cluster? It would be massively useful for our use-case.
Ok, let me know, as otherwise we need to look for other reasons. If the peers are connected, the peerstore file should definitely be written and contain the addresses of the other peer.
Also, why isn’t there an endpoint in the IPFS cluster API to dynamically add peers? Given that this feature is in IPFS https://docs.ipfs.io/reference/api/http/#api-v0-bootstrap-add, is there a reason why is not / can’t be implemented in the IPFS cluster? It would be massively useful for our use-case.
Can you provide more details? Once a peer has joined the cluster it will discover other peers and should remember them.
The only reason I see for such endpoint would be to start a peer as a single-peer cluster and then connecting it to some other peer while running, which is a bit weird because they could have connected to each other on boot with the ‘bootstrap’ flag or the addresses in the peerstore.
But the main reason there is no add is because with Raft, adding a peer is a complicated process and doing it wrong led to bad places. Now that Raft is not the default, I’m happy to revisit this.