Private network p2p-circuit swarm connect success, but ipfs cat not work!

I create a private network.
two bootstrap nodes S1, S2 have the public IP addresses, set RelayServer enabled
two nodes A1, A2 in the local network behind NAT, set RelayClient enabled
two nodes B1, B2 in another local network behind NAT, set RelayClient enabled

All these nodes set up the same swarm.key

S1, S2, A1, and A2 discover each other automatical and work well
S1, S2, B1, and B2 are the same as above.

A1, A2 swarm connect to B1, B2 success via p2p-circuit address
B1, B2 can swarm connect to A1, A2 success via p2p-circuit address too

BUT, A1, A2, and B1, B2 can not get the file with each other.

Why the different local networks can swarm connect to each other via p2p-circuit, but can not exchange data?

> ipfs id 
{
	"ID": "12D3KooWFKQ8jcYyyDo245tFUnSZmMi2yZWmMaHdcUWWGqEYXqSZ",
	"PublicKey": "CAESIFG6gBYsawojrm4i5wqxSy3zNP+7uJvLqUK14jTprPfK",
	"Addresses": [
		"/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWFKQ8jcYyyDo245tFUnSZmMi2yZWmMaHdcUWWGqEYXqSZ",
		"/ip4/192.168.3.93/tcp/4001/p2p/12D3KooWFKQ8jcYyyDo245tFUnSZmMi2yZWmMaHdcUWWGqEYXqSZ",
		"/ip6/::1/tcp/4001/p2p/12D3KooWFKQ8jcYyyDo245tFUnSZmMi2yZWmMaHdcUWWGqEYXqSZ"
	],
	"AgentVersion": "kubo/0.17.0/",
	"ProtocolVersion": "ipfs/0.1.0",
	"Protocols": [
		"/floodsub/1.0.0",
		"/ipfs/bitswap",
		"/ipfs/bitswap/1.0.0",
		"/ipfs/bitswap/1.1.0",
		"/ipfs/bitswap/1.2.0",
		"/ipfs/id/1.0.0",
		"/ipfs/id/push/1.0.0",
		"/ipfs/lan/kad/1.0.0",
		"/ipfs/ping/1.0.0",
		"/libp2p/autonat/1.0.0",
		"/libp2p/circuit/relay/0.1.0",
		"/libp2p/circuit/relay/0.2.0/stop",
		"/libp2p/fetch/0.0.1",
		"/meshsub/1.0.0",
		"/meshsub/1.1.0",
		"/p2p/id/delta/1.0.0",
		"/x/"
	]
}

ipfs id 12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz                                                                                                    
{
	"ID": "12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz",
	"PublicKey": "CAESIKZQt3PWA26ZtLvDLoQWd0s3fq3l/E7lHRmVRaq38PKZ",
	"Addresses": [
		"/ip4/<publicIP_1>/tcp/4001/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit/p2p/12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz",
		"/ip4/<publicIP_1>/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit/p2p/12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz",
		"/ip4/<publicIP_2>/tcp/4001/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit/p2p/12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz",
		"/ip4/<publicIP_2>/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit/p2p/12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz",
		"/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz",
		"/ip4/192.168.3.152/tcp/4001/p2p/12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz",
		"/ip6/::1/tcp/4001/p2p/12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz"
	],
	"AgentVersion": "kubo/0.16.0/",
	"ProtocolVersion": "ipfs/0.1.0",
	"Protocols": [
		"/floodsub/1.0.0",
		"/ipfs/bitswap",
		"/ipfs/bitswap/1.0.0",
		"/ipfs/bitswap/1.1.0",
		"/ipfs/bitswap/1.2.0",
		"/ipfs/id/1.0.0",
		"/ipfs/id/push/1.0.0",
		"/ipfs/lan/kad/1.0.0",
		"/ipfs/ping/1.0.0",
		"/libp2p/autonat/1.0.0",
		"/libp2p/circuit/relay/0.1.0",
		"/libp2p/circuit/relay/0.2.0/stop",
		"/meshsub/1.0.0",
		"/meshsub/1.1.0",
		"/p2p/id/delta/1.0.0",
		"/x/"
	]
}


ipfs swarm peers
/ip4/<publicIP_1>/tcp/4001/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ
/ip4/<publicIP_2>/tcp/4001/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz
/ip4/<publicIP_2>/tcp/4001/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit/p2p/12D3KooWM1bETTRGNe1QgUQDDcnnKMmeHMUPHLFi8fsutiVaktFz

yes, bootstrap nodes S1 and S2 using public facing IP address, and 4001 port allowed

In the following thread, while trying to help someone with a similar issue, I discovered that relay v2 does not support the use of swarm keys:

1 Like

Thank a lot for your reply @ylempereur
:grin:

This does look bad. I use ipfs/kubo as a library to develop based on ipfs applications, which makes the use of similar applications more restrictive.

The real solution is to use the main net instead of a private net (so, no swarm keys, and a lot more nodes available), and encrypt your data end-to-end, in effect creating a private network within the main net

Yes, I now plan to replace the private network with asymmetric encryption with “age” and then use a public node as a bootstrap node.
I’ll test it later to see how it works

I removed the swarm.key file for all nodes and restart ipfs daemon
With the ipfs id command, I can get the address of each node, including the relay address
However, it is still not possible to get the files of other nodes through IPFS

From the logs, it seems that the node does not actively go to the node that relays the address to look up the CID

ipfs debug log :

2022-12-10T14:14:40.702+0800	DEBUG	blockservice	go-blockservice@v0.4.0/blockservice.go:247	Blockservice: Searching bitswap
2022-12-10T14:14:40.702+0800	DEBUG	bitswap	getter/getter.go:86	Bitswap.GetBlockRequest.Start	{"cid": "QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc"}
2022-12-10T14:14:40.702+0800	INFO	bs:sess	session/session.go:460	No peers - broadcasting	{"session": 2, "want-count": 1}
2022-12-10T14:14:40.702+0800	DEBUG	bs:sess	session/session.go:467	broadcastWantHaves	{"session": 2, "cids": [{"/":"QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc"}]}
2022-12-10T14:14:40.703+0800	DEBUG	bitswap	messagequeue/messagequeue.go:663	sent message	{"type": "WANT_HAVE", "cid": "QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc", "local": "12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq", "to": "12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ"}
2022-12-10T14:14:40.703+0800	DEBUG	bitswap	messagequeue/messagequeue.go:663	sent message	{"type": "WANT_HAVE", "cid": "QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc", "local": "12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq", "to": "12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz"}
2022-12-10T14:14:41.703+0800	DEBUG	bs:sess	session/session.go:467	broadcastWantHaves	{"session": 2, "cids": [{"/":"QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc"}]}
2022-12-10T14:14:41.703+0800	DEBUG	bs:sess	session/session.go:361	FindMorePeers	{"session": 2, "cid": "QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc", "pending": 1}
2022-12-10T14:14:41.705+0800	DEBUG	bitswap	providerquerymanager/providerquerymanager.go:329	New Provider Query on cid: QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc
2022-12-10T14:14:41.706+0800	DEBUG	bitswap	providerquerymanager/providerquerymanager.go:234	Beginning Find Provider Request for cid: QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc
2022-12-10T14:14:41.707+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:479	finding providers	{"cid": "QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc", "mh": "bciqpebzxtccy2hi355tgdkumkndlnkqposypb5lfejyjnjq63gmfsgy"}
2022-12-10T14:14:41.708+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:479	finding providers	{"cid": "QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc", "mh": "bciqpebzxtccy2hi355tgdkumkndlnkqposypb5lfejyjnjq63gmfsgy"}
2022-12-10T14:14:41.738+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:541	1 provider entries
2022-12-10T14:14:41.738+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:541	1 provider entries
2022-12-10T14:14:41.738+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:546	got provider: {12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n: [/ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/127.0.0.1/udp/4001/quic /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit /ip4/192.168.3.26/tcp/4001 /ip4/192.168.3.26/udp/4001/quic /ip6/::1/udp/4001/quic]}
2022-12-10T14:14:41.738+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:548	using provider: {12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n: [/ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/127.0.0.1/udp/4001/quic /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit /ip4/192.168.3.26/tcp/4001 /ip4/192.168.3.26/udp/4001/quic /ip6/::1/udp/4001/quic]}
2022-12-10T14:14:41.738+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:563	got closer peers: 1 [{12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ: [/ip4/10.0.0.2/udp/4001/quic /ip4/127.0.0.1/udp/4001/quic /ip6/::1/udp/4001/quic /ip4/10.0.0.2/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/1.14.102.100/tcp/4001 /ip4/1.14.102.100/udp/4001/quic]}]
2022-12-10T14:14:41.738+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/dht.go:646	peer found	{"peer": "12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz"}
2022-12-10T14:14:41.739+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:546	got provider: {12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n: [/ip4/192.168.3.26/tcp/4001 /ip4/192.168.3.26/udp/4001/quic /ip4/127.0.0.1/udp/4001/quic /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit /ip6/::1/udp/4001/quic /ip6/::1/tcp/4001 /ip4/127.0.0.1/tcp/4001]}
2022-12-10T14:14:41.739+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:563	got closer peers: 1 [{12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz: [/ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/10.0.0.11/udp/4001/quic /ip4/127.0.0.1/udp/4001/quic /ip6/::1/udp/4001/quic /ip4/1.14.59.205/tcp/4001 /ip4/1.14.59.205/udp/4001/quic /ip4/10.0.0.11/tcp/4001]}]
2022-12-10T14:14:41.739+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/dht.go:646	peer found	{"peer": "12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ"}
2022-12-10T14:14:41.740+0800	DEBUG	basichost	basic/basic_host.go:718	host 12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq dialing 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n
2022-12-10T14:14:41.740+0800	DEBUG	swarm2	swarm/swarm_dial.go:243	dialing peer	{"from": "12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq", "to": "12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n"}
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:193	[limiter] adding a dial job through limiter: /ip4/192.168.3.26/udp/4001/quic
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:167	[limiter] executing dial; peer: 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/192.168.3.26/udp/4001/quic; FD consuming: 0; waiting: 0
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:193	[limiter] adding a dial job through limiter: /ip4/192.168.3.26/tcp/4001
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:161	[limiter] taking FD token: peer: 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/192.168.3.26/tcp/4001; prev consuming: 0
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:167	[limiter] executing dial; peer: 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/192.168.3.26/tcp/4001; FD consuming: 1; waiting: 0
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:193	[limiter] adding a dial job through limiter: /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:167	[limiter] executing dial; peer: 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit; FD consuming: 1; waiting: 0
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/swarm_dial.go:476	12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq swarm dialing 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n /ip4/192.168.3.26/udp/4001/quic
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:193	[limiter] adding a dial job through limiter: /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:167	[limiter] executing dial; peer: 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit; FD consuming: 1; waiting: 0
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:193	[limiter] adding a dial job through limiter: /ip4/1.14.102.100/tcp/4001/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/limiter.go:167	[limiter] executing dial; peer: 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/1.14.102.100/tcp/4001/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit; FD consuming: 1; waiting: 0
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/swarm_dial.go:476	12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq swarm dialing 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n /ip4/192.168.3.26/tcp/4001
2022-12-10T14:14:41.743+0800	DEBUG	swarm2	swarm/swarm_dial.go:476	12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq swarm dialing 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n /ip4/1.14.102.100/tcp/4001/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit
2022-12-10T14:14:41.744+0800	DEBUG	swarm2	swarm/swarm_dial.go:476	12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq swarm dialing 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit
2022-12-10T14:14:41.745+0800	DEBUG	swarm2	swarm/swarm_dial.go:476	12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq swarm dialing 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit
2022-12-10T14:14:41.745+0800	DEBUG	p2p-circuit	client/dial.go:119	dialing peer 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n through relay 12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ
2022-12-10T14:14:41.745+0800	DEBUG	swarm2	swarm/swarm.go:348	[12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq] opening stream to peer [12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ]
2022-12-10T14:14:41.764+0800	DEBUG	p2p-circuit	client/dial.go:119	dialing peer 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n through relay 12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz
2022-12-10T14:14:41.764+0800	DEBUG	swarm2	swarm/limiter.go:100	[limiter] freeing peer token; peer 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit; active for peer: 5; waiting on peer limit: 0
2022-12-10T14:14:41.764+0800	DEBUG	swarm2	swarm/swarm.go:348	[12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq] opening stream to peer [12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz]
2022-12-10T14:14:41.764+0800	DEBUG	swarm2	swarm/limiter.go:100	[limiter] freeing peer token; peer 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/1.14.102.100/tcp/4001/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit; active for peer: 4; waiting on peer limit: 0
2022-12-10T14:14:41.898+0800	DEBUG	swarm2	swarm/limiter.go:100	[limiter] freeing peer token; peer 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit; active for peer: 3; waiting on peer limit: 0
2022-12-10T14:14:41.898+0800	DEBUG	swarm2	swarm/limiter.go:201	[limiter] clearing all peer dials: 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n
2022-12-10T14:14:41.898+0800	DEBUG	swarm2	swarm/limiter.go:73	[limiter] freeing FD token; waiting: 0; consuming: 1
2022-12-10T14:14:41.899+0800	DEBUG	swarm2	swarm/limiter.go:100	[limiter] freeing peer token; peer 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/192.168.3.26/tcp/4001; active for peer: 2; waiting on peer limit: 0
2022-12-10T14:14:41.900+0800	DEBUG	swarm2	swarm/limiter.go:100	[limiter] freeing peer token; peer 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n; addr: /ip4/192.168.3.26/udp/4001/quic; active for peer: 1; waiting on peer limit: 0
2022-12-10T14:14:41.900+0800	DEBUG	basichost	basic/basic_host.go:416	protocol negotiation took 244.25µs
2022-12-10T14:14:41.901+0800	DEBUG	net/identify	identify/id.go:412	/ipfs/id/1.0.0 sent message to 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit
2022-12-10T14:14:41.936+0800	DEBUG	net/identify	identify/id.go:444	/ipfs/id/1.0.0 received message from 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit
2022-12-10T14:14:41.938+0800	DEBUG	net/identify	identify/id.go:637	12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq received listen addrs for 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n: [/ip4/192.168.3.26/udp/4001/quic /ip4/192.168.3.26/tcp/4001 /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit]
2022-12-10T14:14:41.939+0800	DEBUG	basichost	basic/basic_host.go:735	host 12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq finished dialing 12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n
2022-12-10T14:14:41.939+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/dht.go:663	peer stopped dht	{"peer": "12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n"}
2022-12-10T14:14:41.939+0800	DEBUG	bitswap	providerquerymanager/providerquerymanager.go:329	Received provider (12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n) for cid (QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc)
2022-12-10T14:14:41.939+0800	DEBUG	bitswap	providerquerymanager/providerquerymanager.go:329	Finished Provider Query on cid: QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc
2022-12-10T14:14:41.939+0800	DEBUG	bs:sprmgr	sessionpeermanager/sessionpeermanager.go:68	Bitswap: Added peer to session	{"session": 2, "peer": "12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n", "peerCount": 1}
2022-12-10T14:14:41.940+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/dht.go:663	peer stopped dht	{"peer": "12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n"}
2022-12-10T14:14:42.329+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:479	finding providers	{"cid": "bafkreiajeob2wnx7yzh7sjdocj2heqjlcifcs4ev6g726tiykvbj7u3c7a", "mh": "bciqasi4dvm3p7rsp7esg4etuojasweqkffyjl4n7v5grqvkct7jwf6a"}
2022-12-10T14:14:42.330+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:479	finding providers	{"cid": "bafkreiajeob2wnx7yzh7sjdocj2heqjlcifcs4ev6g726tiykvbj7u3c7a", "mh": "bciqasi4dvm3p7rsp7esg4etuojasweqkffyjl4n7v5grqvkct7jwf6a"}
2022-12-10T14:14:42.356+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:541	1 provider entries
2022-12-10T14:14:42.356+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:541	1 provider entries
2022-12-10T14:14:42.356+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:546	got provider: {12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq: [/ip4/1.14.59.205/tcp/4001/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit /ip4/192.168.3.93/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/192.168.3.93/udp/4001/quic /ip4/127.0.0.1/udp/4001/quic /ip6/::1/udp/4001/quic /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit /ip4/1.14.102.100/tcp/4001/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit]}
2022-12-10T14:14:42.356+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:563	got closer peers: 1 [{12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ: [/ip4/1.14.102.100/tcp/4001 /ip4/1.14.102.100/udp/4001/quic /ip4/10.0.0.2/udp/4001/quic /ip4/127.0.0.1/udp/4001/quic /ip6/::1/udp/4001/quic /ip4/10.0.0.2/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001]}]
2022-12-10T14:14:42.356+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/dht.go:646	peer found	{"peer": "12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz"}
2022-12-10T14:14:42.356+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:546	got provider: {12D3KooWHSXA63GotpYjBW4JvMvfCY8X4U57G2TgdUztdTcYvUjq: [/ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/192.168.3.93/udp/4001/quic /ip4/127.0.0.1/udp/4001/quic /ip6/::1/udp/4001/quic /ip4/1.14.59.205/tcp/4001/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit /ip4/192.168.3.93/tcp/4001 /ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit /ip4/1.14.102.100/tcp/4001/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit /ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ/p2p-circuit]}
2022-12-10T14:14:42.357+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/routing.go:563	got closer peers: 1 [{12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz: [/ip6/::1/udp/4001/quic /ip4/1.14.59.205/tcp/4001 /ip4/1.14.59.205/udp/4001/quic /ip4/10.0.0.11/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/10.0.0.11/udp/4001/quic /ip4/127.0.0.1/udp/4001/quic]}]
2022-12-10T14:14:42.357+0800	DEBUG	dht	go-libp2p-kad-dht@v0.18.0/dht.go:646	peer found	{"peer": "12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ"}
2022-12-10T14:14:42.704+0800	DEBUG	bs:sess	session/session.go:467	broadcastWantHaves	{"session": 2, "cids": [{"/":"QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc"}]}
2022-12-10T14:14:44.705+0800	DEBUG	bs:sess	session/session.go:467	broadcastWantHaves	{"session": 2, "cids": [{"/":"QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc"}]}
2022-12-10T14:14:47.707+0800	DEBUG	bs:sess	session/session.go:467	broadcastWantHaves	{"session": 2, "cids": [{"/":"QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc"}]}
2022-12-10T14:14:51.708+0800	DEBUG	bs:sess	session/session.go:467	broadcastWantHaves	{"session": 2, "cids": [{"/":"QmedRkm5VMLTpeLJ8BhtbLTFNzJAP4vs7K9JXehxpkKoqc"}]}

when run ipfs swarm peers
the node behind NAT circuit address showed, but can not cat file

>  ipfs swarm peers                                                                                                                                                            zeta@MacBook-Pro
/ip4/1.14.102.100/udp/4001/quic/p2p/12D3KooWBBbdgzJBLUUFhMpA9JucE932wJNt2d6QZrGgSmPvTtPZ
/ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz
/ip4/1.14.59.205/udp/4001/quic/p2p/12D3KooWCjiGPnmxpsZnH1Zv2DzHJ5ReigdJhCsbnvB2ZXdjQrvz/p2p-circuit/p2p/12D3KooWB9iskrK9VoLYF2DgR7CRVSGL8f3hv1SHmqARv9ud1o2n

Connecting using a relay v2 is a two step process. First, it connects through the relay itself and communicates using that connection for simple things (low bandwidth). Second, if it wants to use bitswap (to exchange blocks), it tries to connect directly using hole punching, and then uses that connection for bitswap (the relay connection is eventually discarded, as it’s no longer needed).

If for some reason it’s not able to establish a hole punch connection, bitswap can never work (and therefore, no cat). So, there must be a direct path between the two nodes, with NAT being the only problem preventing them from connecting (which is what hole punching gets around, but it has a fickle timing and can fail. there’s a whole round of public testing of it going on this month to try and make it more reliable).

Thanks a lot ! @ylempereur :grin:
As you said, the nodes behind NAT communicate very unstable, when I use ipfs mainnet, nodes behind different NATs can occasionally exchange data with each other, but it is very unstable and sometimes disconnects during communication.
It would be exciting if this issue could be resolved soon, as such use cases are very common and can greatly increase the motivation and interest of developers to use IPFS to develop applications.

I sovled it , when I add 4 public ip bootstrap nodes ,it works!

1 Like