Pubsub not working for remote nodes unless intermediate peer is subscribed to topic

Help Kubo
,
1

Hi,
I have a VM running in the cloud running the Kubo/ipfs-go daemon with a private network, i.e. a private swarm.key.

Now, I am trying to run my own go program (in Docker containers, orchestrated by Kubernetes), that uses Kubo, and want to publish something on one of my laptops and want to subscribe and receive the published message on another of my laptops (both using the private swarm.key). I initially connect my programs to the cloud node and can successfully publish messages to the node. I can subscribe to the topic on my cloud node by running ipfs pubsub sub test and receive all the published messages from both my laptops, and I also can subscribe to the topic on both my laptops and receive each others messages. However, when I stop subscribing to the topic on the cloud node, both my laptops stop receiving the messages of the other laptop.

I have read that in order for pubsub to work, all intermediate peers need to be subscribed to the topic. For that reason I tried to connect the two laptop nodes using a p2p-circuit, i.e. by connecting to ip4/<CLOUD NODE IP>/tcp/4001/p2p/12...<CLOUD>/p2p-circuit/p2p/Qm...<Laptop1/Laptop2>. To my understanding this should be able to circumvent any firewalls in my local network, as the messages are just relayed through the cloud node (please correct me if I missunderstood). However, when I try this in my go programs, I get this error: error opening relay circuit: HOP_CANT_SPEAK_RELAY (270).

When I run ipfs swarm peers on my cloud node I see both my programs connected, e.g.:

# ipfs swarm peers
/ip4/<MY IP>/tcp/49245/p2p/QmRpe3WHYbXn3vRv7NHzByzpNRcoLsGqWV6A6Mre9E6xuS
/ip4/<MY IP>/tcp/49275/p2p/QmZqm243DpJJCm6LQZPSznh7sVAGvy29o1iYDvW47uEkeN

However, when I list all the peers that my programs are connected using api.Swarm().Peers(ctx) I only get my cloud node.

When I tried to run two docker containers in the same Kubernetes cluster on the same machine, I did not need to subscribe to the topic on the cloud node, and both instances received each others messages. However, if I run my software on two different laptops and thus two seperate Kubernetes clusters, I do not receive the messages. I also tried to run my software in separate Kubernetes clusters in the cloud, but I still did not receive the messages of the other instances.

Can someone please give me some advice how I can send messages using IPFS pubsub from one machine to another one without having my cloud node having to subscribe to each of the topics? Maybe I am just missing some cruicial settings? Thank you! :slight_smile:

I am using v0.14.0, but also tried v0.13.0. For my go program, appart from the default config, I use the following settings for creating the repo:

cfg, err := config.Init(ioutil.Discard, 4096)
if err != nil {
	return nil, err
}

cfg.Bootstrap = []string{}
cfg.Discovery.MDNS.Enabled = true
cfg.Routing.Type = config.NewOptionalString("dhtserver")
cfg.Pubsub.Enabled = config.True
cfg.Pubsub.Router = "gossipsub"
cfg.Pubsub.DisableSigning = true
cfg.Ipns.UsePubsub = config.True
cfg.Swarm.RelayService = config.RelayService{
	Enabled: config.True,
}
cfg.Swarm.RelayClient = config.RelayClient{
	Enabled: config.True,
}
cfg.Swarm.Transports.Network.Relay = config.True

and the following settings for creating a node:

node, err := core.NewNode(ctx, &core.BuildCfg{
	Online:    true,
	Routing:   libp2p.DHTServerOption,
	Repo:      repo,
	Permanent: true,
	ExtraOpts: map[string]bool{
		"pubsub": true,
	},
})

My cloud node has this config:

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "GET",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "9GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12...<CLOUD>"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128,
    "UsePubsub": true
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": true,
    "Enabled": true,
    "Router": "gossipsub"
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "dhtserver"
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {
      "GracePeriod": "20s",
      "HighWater": 900,
      "LowWater": 600,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "EnableHolePunching": true,
    "RelayClient": {
      "Enabled": true
    },
    "RelayService": {
      "Enabled": true
    },
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {
        "Relay": true
      },
      "Security": {}
    }
  }
}
1 Like
2

I was now able to connect my two private computers by adding a second node in the cloud. According to this thread: How to setup V1 Relay in the new config there need to be two nodes in order for the relay service in the cloud to start so that other nodes can use a p2p-circuit.

However, I am still not able to publish messages on one of my private computers and receive them on my other private computer, without having at least one of the cloud nodes being subscribed to the topic aswell. If any of the cloud nodes is subscribed to the topic, my other private computer receives the messages, if non of the cloud nodes are subscribed, my private computer does not receive any messages.

Any ideas how I can receive messages on my private computers without having cloud nodes that are subscribed to the topic? Thank you!

3

I can see that my private computers are connected via a p2p-circuit if I run api.Swarm().Peers(ctx). However, if I run api.PubSub().Peers(ctx) only the cloud nodes are listed.

How can I get my private computers node IDs to also be part of the pubsub peers?

4

I get the same problem .
local network work well, but A network can not connect to B network, except bootstrap sub the same topic .

5

I was not able to solve my problem. I can add files using IPFS in network A, and get them in network B, but I cannot publish a message to a topic in network A, and receive the message in network B, unless any of my relay nodes are also actively subscribed to the same topic.

6

I solved this issue .
I add other bootstrap , when 4 bootstrap with public IP, it works fine!

1 Like