Missing public multiaddresses in peerstore (docker)

Hello,

My setup is as follows:

Two separated VPS, running docker with public addresses. I deployed one cluster and one IPFS instance in each, based on the docker compose provided. I want to connect both using the public IP, not the internal docker one.

When adding manually one of the peers in the peerstore, they connect each other, but after shutting down, only the public address provided manually is saved. In other words, when listing the peers:

/ # ipfs-cluster-ctl  peers ls
12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj | cluster0 | Sees 1 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/p2p/12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj
    - /ip4/172.20.0.3/tcp/9096/p2p/12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj
  > IPFS: QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/104.248.37.115/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/127.0.0.1/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/172.20.0.2/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9 | cluster0 | Sees 1 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/p2p/12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9
    - /ip4/172.18.0.3/tcp/9096/p2p/12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9
  > IPFS: QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/127.0.0.1/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/138.68.106.96/tcp/1161/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/138.68.106.96/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/172.18.0.2/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw

As you can see, in the IPFS part the public IP multiaddress is listed, but for the cluster is not. So when one of the nodes restarts, the public ip of the other peer is lost. Is this a limitation of the docker setup? Why is it listed in IPFS then?

the compose relevant part:

  cluster0:
    container_name: cluster0
    image: ipfs/ipfs-cluster:latest
    depends_on:
      - ipfs0
    environment:
      CLUSTER_PEERNAME: cluster0
      CLUSTER_SECRET: ${CLUSTER_SECRET} # From shell variable if set
      CLUSTER_IPFSHTTP_NODEMULTIADDRESS: /dns4/ipfs0/tcp/5001
      CLUSTER_CRDT_TRUSTEDPEERS: '*' # Trust all peers in Cluster
      CLUSTER_RESTAPI_HTTPLISTENMULTIADDRESS: /ip4/0.0.0.0/tcp/9094 # Expose API
      CLUSTER_MONITORPINGINTERVAL: 2s # Speed up peer discovery
    ports:
          # Open API port (allows ipfs-cluster-ctl usage on host)
          - "127.0.0.1:9094:9094"
          # The cluster swarm port would need  to be exposed if this container
          # was to connect to cluster peers on other hosts.
          # But this is just a testing cluster.
          - "9096:9096" # Cluster IPFS Proxy endpoint

This sounds like this issue Peerstore persistence.

I think:

  • The cluster peers are not aware of their external IP
  • Therefore they are not announcing it during identify()
  • Therefore it is not part of the peerstore of the other peer
  • Therefore it is not saved on shutdown.

The thing is Identify() will only start advertising one of those addresses when it has seen at least 4 connections in the last hour (which is the signal that such address is actually a public one). Of course, the thousands of nodes in the ipfs network trigger such condition pretty easily and your ipfs nodes show the public IP, but having only 2 cluster peers this never happens.

I’ll open an issue to reduce the threshold. The workaround is to write your public addresses manually in the peerstore for the moment. If they change, I recommend using /dns4//… form if possible.

Nice catch @hector! Is there any workaround for a node to force to be aware of its own public IP provided I (or another process in the same host) already knows it?

An ideal scenario would be to provide an ENV to the cluster process such as OWN_PUBLIC_IP or something alike so it can advertise this info to other peers

Unfortunately we have no way to foce announcing a user-given address, but let’s fix that along with the issue…

Hey @dapplion, there is a “latest” ipfs/ipfs-cluster docker image from today that enables a bunch of NAT-hole-punching things. Can you test it out? It’s not a specific fix for this, but maybe as a side-effect it gets what you want.

Hi @hector , I work with @dapplion at DAppNode, we are both working on this.

I crated a new environment, pulling latest images, with 4 ipfs-cluster instances, 2 on each host, and I still can’t see any public address listed:

12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj | cluster0 | Sees 3 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/p2p/12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj
    - /ip4/172.20.0.3/tcp/9096/p2p/12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj
  > IPFS: QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/104.248.37.115/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/127.0.0.1/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/172.20.0.2/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9 | cluster0 | Sees 3 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/p2p/12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9
    - /ip4/172.18.0.3/tcp/9096/p2p/12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9J
  > IPFS: QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bwH
    - /ip4/127.0.0.1/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bwm
    - /ip4/138.68.106.96/tcp/1163/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/172.18.0.2/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bwe
12D3KooWGQhJbkabVfjT17GXn2B92MMVtc7xonMpxmdZGsHCTJ91 | cluster1 | Sees 3 other peers 
  > Addresses:.
    - /ip4/127.0.0.1/tcp/9096/p2p/12D3KooWGQhJbkabVfjT17GXn2B92MMVtc7xonMpxmdZGsHCTJ91C
    - /ip4/127.0.0.1/udp/9096/quic/p2p/12D3KooWGQhJbkabVfjT17GXn2B92MMVtc7xonMpxmdZGsHCTJ91
    - /ip4/172.18.0.4/tcp/9096/p2p/12D3KooWGQhJbkabVfjT17GXn2B92MMVtc7xonMpxmdZGsHCTJ91C
    - /ip4/172.18.0.4/udp/9096/quic/p2p/12D3KooWGQhJbkabVfjT17GXn2B92MMVtc7xonMpxmdZGsHCTJ91
  > IPFS: QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bwH
    - /ip4/127.0.0.1/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bww
    - /ip4/138.68.106.96/tcp/1163/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/172.18.0.2/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
12D3KooWPDA1TvisGD3fvHnv3yb9nt693ph7UXhGm6U4Zg9PJmP6 | cluster1 | Sees 3 other peersP
  > Addresses:.
    - /ip4/127.0.0.1/tcp/9096/p2p/12D3KooWPDA1TvisGD3fvHnv3yb9nt693ph7UXhGm6U4Zg9PJmP66
    - /ip4/127.0.0.1/udp/9096/quic/p2p/12D3KooWPDA1TvisGD3fvHnv3yb9nt693ph7UXhGm6U4Zg9PJmP66
    - /ip4/172.20.0.4/tcp/9096/p2p/12D3KooWPDA1TvisGD3fvHnv3yb9nt693ph7UXhGm6U4Zg9PJmP6
    - /ip4/172.20.0.4/udp/9096/quic/p2p/12D3KooWPDA1TvisGD3fvHnv3yb9nt693ph7UXhGm6U4Zg9PJmP6
  > IPFS: QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hbC
    - /ip4/104.248.37.115/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/127.0.0.1/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/172.20.0.2/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb

Hi @vdo, @dapplion, can you test with the docker image ipfs/ipfs-cluster:949test and report back?

It works now! I can see the public multiaddresses, and they are stored in the peerstore at shutdown:

docker-compose exec cluster0 ipfs-cluster-ctl peers ls
WARNING: The CLUSTER_SECRET variable is not set. Defaulting to a blank string.
12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj | cluster0 | Sees 1 other peers
  > Addresses:
    - /ip4/104.248.37.115/tcp/9096/p2p/12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj
    - /ip4/127.0.0.1/tcp/9096/p2p/12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj
    - /ip4/172.22.0.3/tcp/9096/p2p/12D3KooWANNpjTJ4ZDKYVHyknHATQn1pfSvRBpcvk9ty5HGzFNRj
  > IPFS: QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/104.248.37.115/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/127.0.0.1/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
    - /ip4/172.22.0.2/tcp/4001/p2p/QmcYMCpcG6ZCjTm7nrtDGBCRZDaR1WU2BGM8U4WMNGQ8hb
12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9 | cluster0 | Sees 1 other peers
  > Addresses:
    - /ip4/127.0.0.1/tcp/9096/p2p/12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9
    - /ip4/138.68.106.96/tcp/9096/p2p/12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9
    - /ip4/172.18.0.3/tcp/9096/p2p/12D3KooWAui33wf4ruL7VW8iXPPpuheMuCufjL97zeEWQU3H7ua9
  > IPFS: QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/127.0.0.1/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/138.68.106.96/tcp/1163/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/138.68.106.96/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw
    - /ip4/172.18.0.2/tcp/4001/p2p/QmZVHyhqsxKKmS59ZHvt3HHQh46paLDP84Jw4D8GY9Q9bw

Is this a fix that can be merged ?

Thanks!

Awesome thank you so much @hector

I have to make sure this not making things worse for nat’ed hosts or has some other pitfalls, but I think we can merge it…

We merged the fix so it will come along with the next release, but you can also use the latest docker image.

Thank you so much for resolving this issue quickly.