Using Relay in private swarm for punch hole in 0.18

Hello,

I have problem with the Circuit Relay on IPFS 0.18
I’m trying to make 2 undialable node speak to one another by using a dialable node that will act as a Relay. All these 3 nodes have a private swarm key, they are on our private IPFS network.

From what I saw, my relay seems to have trouble using the hop protocol to connect to the relay.
This is the output of the ipfs id of the relay node.

{
	"ID": "12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
	"PublicKey": "CAESIHGHCYeBA4dxwPxJePs0OyocuS7eMJhVjbrGtKQzWeEP",
	"Addresses": [
		"/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/127.0.0.1/tcp/4004/ws/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/127.0.0.1/tcp/80/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/127.0.0.1/udp/4001/quic/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/MY_EXTERNAL_IP/tcp/4001/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/MY_EXTERNAL_IP/tcp/80/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/MY_EXTERNAL_IP/udp/4001/quic/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/172.29.0.3/tcp/4001/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
		"/ip4/172.29.0.3/tcp/4004/ws/p2p/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn"
	],
	"AgentVersion": "kubo/0.18.0/6750377",
	"ProtocolVersion": "ipfs/0.1.0",
	"Protocols": [
		"/floodsub/1.0.0",
		"/ipfs/bitswap",
		"/ipfs/bitswap/1.0.0",
		"/ipfs/bitswap/1.1.0",
		"/ipfs/bitswap/1.2.0",
		"/ipfs/id/1.0.0",
		"/ipfs/id/push/1.0.0",
		"/ipfs/lan/kad/1.0.0",
		"/ipfs/ping/1.0.0",
		"/libp2p/autonat/1.0.0",
		"/libp2p/circuit/relay/0.1.0",
		"/libp2p/circuit/relay/0.2.0/stop",
		"/p2p/id/delta/1.0.0",
		"/x/"
	]
}

This is the config of the relay node : (I tried bootstraping and peering to myself bc I saw someone do this and claiming it help, but it doesn’t do anything for me)

{
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/%%API_PORT%%",
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "Swarm": [
      "/ip4/0.0.0.0/tcp/%%SWARM_PORT%%",
      "/ip4/0.0.0.0/udp/%%SWARM_PORT%%",
      "/ip6/::/tcp/%%SWARM_PORT%%",
	    "/ip4/0.0.0.0/tcp/%%WS_PORT%%/ws"
    ],
    "Announce": [],
    "AppendAnnounce": [
      "/ip4/127.0.0.1/tcp/80",
      "/ip4/127.0.0.1/tcp/4001",
    	"/ip4/127.0.0.1/udp/4001/quic",
	    "/ip4/MY_EXTERNAL_IP/tcp/80",
	    "/ip4/MY_EXTERNAL_IP/tcp/4001",
	    "/ip4/MY_EXTERNAL_IP/udp/4001/quic"
    ],
    "NoAnnounce": []
  },
  "API": {
    "HTTPHeaders": {
        "Access-Control-Allow-Origin": [
                "*"
        ]
    }
  },
  "AutoNAT": {
    "ServiceMode": "enabled",
    "Throttle": {
      "GlobalLimit": 30,
      "PeerLimit": 3
    }
  },
  "Bootstrap": [
    "/ip4/MY_EXTERNAL_IP/tcp/80/ipfs/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
    "/ip4/MY_EXTERNAL_IP/udp/80/ipfs/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
    "/ip4/MY_EXTERNAL_IP/tcp/4001/ipfs/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
    "/ip4/MY_EXTERNAL_IP/udp/4001/ipfs/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn"
  ],
  "Datastore": {
    "StorageMax": "",
    "StorageGCWatermark": 0,
    "GCPeriod": "",
    "HashOnRead": false,
    "BloomFilterSize": 0,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    }
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false
    }
  },
  "Gateway": {
    "NoFetch": false,
    "NoDNSLink": false,
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "%%SWARM_ID%%",
    "PrivKey": "%%PRIVATE_KEY%%"
  },
  "Ipns": {
    "RepublishPeriod": "",
    "RecordLifetime": "",
    "ResolveCacheSize": 128,
    "UsePubsub": false
  },
  "Migration": {
    "DownloadSources": ["HTTPS", "IPFS"],
    "Keep": "cache"
  },
  "Mounts": {
    "IPFS": "/ipfs",
    "IPNS": "/ipns",
    "FuseAllowOther": false
  },
  "PubSub": {
    "Enabled": true,
    "Router": "floodsub",
    "DisableSigning": false
  },
  "Peering": {
    "Peers": [
      {
        "ID": "12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn",
        "Addrs": [
          "/ip4/152.228.229.46/tcp/80",
          "/ip4/152.228.229.46/udp/80",
          "/ip4/152.228.229.46/tcp/4001",
          "/ip4/152.228.229.46/udp/4001"
        ]
      }
    ]
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": null,
    "DisableBandwidthMetrics": true,
    "DisableNatPortMap": false,
    "EnableHolePunching": true,
    "RelayClient": {
      "Enabled": true,
      "StaticRelays": [
      ]
    },
    "RelayService": {
      "Enabled": true,
      "ConnectionDurationLimit": "2m",
      "ConnectionDataLimit": 131072,
      "ReservationTTL": "1h",
      "MaxReservations": 128,
      "MaxCircuits": 16,
      "BufferSize": 2048,
      "MaxReservationsPerPeer": 4,
      "MaxReservationsPerIP": 8,
      "MaxReservationsPerASN": 32
    },
    "ConnMgr": {
      "Type": "basic",
      "GracePeriod": "20s",
      "HighWater": 900,
      "LowWater": 600
    }
  }
}

The config of the non-dialable node are basically the same with different key, and with :

    "RelayClient": {
      "Enabled": true,
      "StaticRelays": [
        "/p2p-circuit/ipfs/12D3KooWHTXfbZWVVJZBe4efvL2atLooRhC35f9u8J4hAqgggvYn"
      ]
    },

In order to try to see if the problem was during the discovery of other peers during Kademlia protocol.

My 2 un-dialable nodes :

  • can’t communicate together
  • can’t ping one another
  • when I add a file in IPFS in node A, node B can’t cat it unless the relay server cat it first

So it seems that the relay circuit have a problem when using the hop protocol. Do you have an idea of what could be happening ? Is it something else ?

I solved my problem by adding another node that allows the relay node to find another peer to advertise itself as a p2p-circuit.

Now, i’m able to connect from A to B by the relay either with ipfs swarm connect or with the autodiscovery, but I’m still not able to do ipfs add on A and retrieve it with ipfs cat on B

What am I missing ? Is the hole punching failing ? How can I verify that the hole punching has suceed ?

up ? Anyone having an idea ?

Is relay and hole punching even possible with private swarm key ?

Try a complete address in the StaticRelays, also /ipfs/ in multiaddrs has been deprecated for a while now (just use /p2p/ instead).

Things to try:

  • Can you see the relayed address when you do ipfs id on the undialable nodes ?
  • Does ipfs swarm connect $RELAY_ADDR/p2p-circuit/p2p/... on one of the undialable nodes print connect success ?

I can see the relayed adress in ipfs id and the connection with p2p-circuit works fine, but i’m not able to do the following sequence even when the connection is made to the relay R :

  • ipfs add of random string in .txt on an undialable node A → Success
  • ipfs cat of the CID of the previous ipfs add on undialable node B → Keep hanging

I suspect that the hole punching is not correctly working, i’m able to do ipfs ping from A to B, i’m able to find B from A with ipfs dht findpeer, but not receiving data apparently ?

dht findpeer does not require establishing a connection, it only do a dht lookup (and there the relays knows everyone).

Does that works ? (manually connecting from B → A using ipfs swarm connect using the relay address you are seeing on A) :slightly_smiling_face:

Yes, as I said I am able to connect with p2p-circuit and it works fine, i can see the p2p-circuit adress in ipfs swarm peers after connection

Just wanted to be sure, so that is likely that your holepunching attempts fails (the connection success message isn’t very accurate, it looks for any connection, including relays, even tho bitswap refuse to run over relays.)

I saw somewhere someone saying that holepunching wasn’t available with private swarm keys, is it true ? If yes do you know if it’s planned to have this feature ?

Up, anyone having an idea ?