Auto discovery with multiple docker networks: only one network is listed in "swarm addrs local"

what works

  • ipfs nodes do auto discover each other on the same single docker network as expected
  • each IPFS container’s respective internal IP address is listed both in swarm addrs listen and swarm addrs local

problem

  • when we add (in docker-compose.yml) another docker network for IPFS node (now it has 2), we see both IP addresses (for each network) in swarm addrs listen as expected, but - only one gets into swarm addrs local
  • same applies to more than 2 networks: all listed in listen, but only one gets into local
  • without IP being in swarm addrs local nodes cannot auto discover ( obviously, expected behavior)

why

  • the project is developing a multiplayer game
  • each player’s app instance is a docker-compose stack/deployment
  • services within docker-compose are naturally on a default network to communicate
  • but: we add another network specifically for IPFS nodes to share so they could auto discover themselves while being in different stacks/deployments
  • looks something like
  • example output (below) for one of the stacks
    • for some reason, /ip4/172.19.0.2/tcp/4001 address is missing from local, and it’s exactly the one from IPFS nodes shared docker network
    • dialing from IPFS node (which is also on 172.19.x.x. network) but on another stack/deployment works: nc 172.19.0.2 4001 shows multistream 1.0, swarm connect /ip4/172.19.0.2/tcp/4001/p2p/... connects successfully, connection persists and pubsub between nodes also works
    • the only problem is: address is missing from local, so no autodiscovery …
user@pc:~/code/DeathStarGame/bin$ dev exec ipfs ipfs swarm addrs listen
/ip4/127.0.0.1/tcp/4001
/ip4/127.0.0.1/udp/4001/quic
/ip4/172.18.0.4/tcp/4001
/ip4/172.18.0.4/udp/4001/quic
/ip4/172.19.0.2/tcp/4001
/ip4/172.19.0.2/udp/4001/quic
/p2p-circuit
user@pc:~/code/DeathStarGame/bin$ dev exec ipfs ipfs swarm addrs local
/ip4/127.0.0.1/tcp/4001
/ip4/127.0.0.1/udp/4001/quic
/ip4/172.18.0.4/tcp/4001
/ip4/172.18.0.4/udp/4001/quic
/ip4/46.x.x.x/tcp/1053
/ip4/46.x.x.x/tcp/4001
/ip4/46.x.x.x/udp/4001/quic
  • *connecting via p2p-circuit works, but still, for development docker networks would be preferable

sum-up question

The list of addresses comes from https://github.com/libp2p/go-libp2p/blob/master/p2p/host/basic/basic_host.go#L773

Using a AddrsFactory that comes from: https://github.com/ipfs/go-ipfs/blob/master/core/node/libp2p/addrs.go#L28

Since both swarm local and swarm listen are based on h.Network().InterfaceListenAddresses(), perhaps it is something in the config (Announce/NoAnnounce) that is getting in the way and filtering out the other local addresses.

Thank you for the reply,

  • as expected, was too cavalier with the code, it requires deep understanding
  • Announce and NoAnnounce are empty, config is completely default, node forms the local and listen automatically, which is great
  • could I have missed something in the docs that makes it possible to influence swarm addrs local ?
    (AFAIK, If we put anything into Announce, it overrides completely, without merging with auto generated addresses)
/data/ipfs/config
{
"API": {
  "HTTPHeaders": {}
},
"Addresses": {
  "API": "/ip4/0.0.0.0/tcp/5001",
  "Announce": [],
  "Gateway": "/ip4/0.0.0.0/tcp/8080",
  "NoAnnounce": [],
  "Swarm": [
    "/ip4/0.0.0.0/tcp/4001",
    "/ip6/::/tcp/4001",
    "/ip4/0.0.0.0/udp/4001/quic",
    "/ip6/::/udp/4001/quic"
  ]
},
"AutoNAT": {},
"Bootstrap": [
  "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
  "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
  "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
  "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
  "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
  "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
],
"Datastore": {
  "BloomFilterSize": 0,
  "GCPeriod": "1h",
  "HashOnRead": false,
  "Spec": {
    "mounts": [
      {
        "child": {
          "path": "blocks",
          "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
          "sync": true,
          "type": "flatfs"
        },
        "mountpoint": "/blocks",
        "prefix": "flatfs.datastore",
        "type": "measure"
      },
      {
        "child": {
          "compression": "none",
          "path": "datastore",
          "type": "levelds"
        },
        "mountpoint": "/",
        "prefix": "leveldb.datastore",
        "type": "measure"
      }
    ],
    "type": "mount"
  },
  "StorageGCWatermark": 90,
  "StorageMax": "10GB"
},
"Discovery": {
  "MDNS": {
    "Enabled": true,
    "Interval": 10
  }
},
"Experimental": {
  "FilestoreEnabled": false,
  "GraphsyncEnabled": false,
  "Libp2pStreamMounting": false,
  "P2pHttpProxy": false,
  "ShardingEnabled": false,
  "StrategicProviding": false,
  "UrlstoreEnabled": false
},
"Gateway": {
  "APICommands": [],
  "HTTPHeaders": {
    "Access-Control-Allow-Headers": [
      "X-Requested-With",
      "Range",
      "User-Agent"
    ],
    "Access-Control-Allow-Methods": [
      "GET"
    ],
    "Access-Control-Allow-Origin": [
      "*"
    ]
  },
  "NoDNSLink": false,
  "NoFetch": false,
  "PathPrefixes": [],
  "PublicGateways": null,
  "RootRedirect": "",
  "Writable": false
},
"Identity": {
  "PeerID": "12D3KooWArVuq9DpGWjwLVNoyL9dwXvK9ZADZSDVJMxFmxbppAdu",
  "PrivKey": "CAESQO8a1+MV1BzCGlkSK0Ya8uFJJ5Nd0siG7JRnjdUgwJGmD2eaequFKOhGeFmbFS1x+g9nxZeE5DCQjuDHAMjx0ug="
},
"Ipns": {
  "RecordLifetime": "",
  "RepublishPeriod": "",
  "ResolveCacheSize": 128
},
"Mounts": {
  "FuseAllowOther": false,
  "IPFS": "/ipfs",
  "IPNS": "/ipns"
},
"Peering": {
  "Peers": null
},
"Plugins": {
  "Plugins": null
},
"Provider": {
  "Strategy": ""
},
"Pubsub": {
  "DisableSigning": false,
  "Router": ""
},
"Reprovider": {
  "Interval": "12h",
  "Strategy": "all"
},
"Routing": {
  "Type": "dht"
},
"Swarm": {
  "AddrFilters": null,
  "ConnMgr": {
    "GracePeriod": "20s",
    "HighWater": 900,
    "LowWater": 600,
    "Type": "basic"
  },
  "DisableBandwidthMetrics": false,
  "DisableNatPortMap": false,
  "EnableAutoRelay": false,
  "EnableRelayHop": false,
  "Transports": {
    "Multiplexers": {},
    "Network": {},
    "Security": {}
  }
}
}

Afaik, it is not possible to specify “additional” addresses that merge with the autodiscovered ones.

I was going to ask you about disabling relay but it’s already disabled in your config. What about disabling MDNS? Does it have any effect?

Hm, seemed counter-intuitive (isn’t MDNS the key to auto discovery?) but I tried anyway , sadly no effect … Launched with ipfs config Discovery.MDNS.Enabled --bool false (config is set before the daemon start, so it’s in action)

To put it even simpler: if within a single docker-compose we define several IPFS nodes and several networks and assign them to every node, every node finds all the networks and lists them in addrs listen - perfect! - but for some reason somewhere a decision is made to pick just one network (randomly?) into addrs local to be advertised… Would be nice to understand that moment.

I do understand there may be a rationale for that I just don’t know about. But even if there is for picking one, it chooses randomly, and that’s not ideal.

No, you’re right , thank you, links to code you gave seem exactly what I’m looking for.
It does seem that in core/node/libp2p/addrs.go#makeAddrsFactory the only factors for filtering are Announce and NoAnnounce

Let me try forking and changing the code, maybe I’ll tweak my way into addrs local mathcing addrs listen (in regards to docker networks)

To sum up current status (using go-ipfs#v0.7.0 and repsectively go-libp2p#v0.11.0 go-libp2p-swarm#0.2.8)

Did fork and run and indeed - makeAddrsFactory https://github.com/ipfs/go-ipfs/blob/v0.7.0/core/node/libp2p/addrs.go#L28 returns go-libp2p-swarm/swarm_addr.go#AllAddrs when Announce/NoAnnounce are empty

Will explore further to make it so that all docker networks are within core/coreapi/swarm#LocalAddrs.
The focus is on AllAddrs, this where IPs get filtered out.
*There is always a monkey-patch option of making core/coreapi/swarm#LocalAddrs return merged InterfaceListenAddresses and AllAddrs, but this is a hack we need to avoid. (would not matter, coreapi/swarm has no effect on actual discovery)

Hmm. I was actually reading an old version of basic_host.go.

I think the logic in AllAddrs() got too clever and this is a bug. Can you open an issue in github.com/libp2p/go-libp2p and ask? libp2p devs should be able to give better insights…

Yes, definitely, will do. Today though will explore a little more )

Created an issue in go-libp2p:

Update:

*will do further comments in github issue