Cannot make P2P connection in IPFS work in real but isolated network(Currently Internet cutoff in Iran) even with custom bootstrap node

Help
1

Hi currently in Iran government did cutoff international internet and Iranian can only access iranian IPs.

I’m trying to setup IPFS bootstrap node so others can join the a private IPFS network(I call it private because it’s only accessable in Iran because as non of nodes have international internet access this network would not be visible to the world. my purpose is allow iranian peaple to send files to each other safely). obviously now official bootstrap node not work so i did:

  • setup `kubo` on my server and did open port in firewall(both 4001 tcp and udp). i did removed all bootstrap nodes using ipfs bootstrap rm --all and ran the server using server profile.
  • in my laptop i removed all bootstrap nodes and added my server to bootstrap node list
  • in my friend laptop i did the same(i did confirm the i can connect to my server and it’s working) and created a file using echo "hello IPFS" | ipfs add -q --cid-version 1
  • tried to fetch the file in my friend laptop using ipfs cat <the_output_of_above_command> and the result was nothing. it freezes and do not show anything
  • i tried fetching the file in the bootstrap server and it worked and ipfs cat result was “hello IPFS”(this means direct server->myLaptop and myLaptop->server works)
  • then i tried the return the same command in my friend laptop and it worked!(which means direct server->myFriendLaptop and myFriendLaptop->server also works)
  • i did the same precedure again with another text message and i get the same result as before(ipfs cat freezes and do not output anything). This means myLaptop ↔ myFriendLaptop connection does not work

I also tried these

  • adding GOLOG_LOG_LEVEL=error,provider=debug,dht/provider=debug to kubo docker container did not helped(i now get some extra logs but not much)
  • confirm that i’m connected to bootstrap node as ipfs swarm connect /dnsaddr/... results was success.
  • i’m sure that UDP port is open. although ISP maybe is blocking it(i will check it today using ncat -u)
  • both three nodes are running the same ipfs/kubo:master-latest image in docker and have volume setuped for persistant config(i did confirm that any change in ipfs config is persistant after container rerun)
2

But are you connected to your friend?

  • You need to be connected to your friend to fetch content from them
  • Autodiscovery of your friend means they need to have a public IP and WAN-DHT working (ipfs id should show "/ipfs/kad/1.0.0", among the protocols). Once that exists they should be announcing their content to the DHT and to you (in a small network like that everyone would be connected to everyone).
  • There is AutoNAT and hole-punching to obtain public IPs, but perhaps in a 3 people network this is not working well or UDP is blocked as you said.
3

I was using docker rootless with default configuration which removes source ip from incoming connection, i fixed it by switching to pasta.

but still i have same problem but this seems that it’s improved compare to before fixing docker network.
i still do not get "/ipfs/kad/1.0.0" in ipfs result, there is a /ipfs/lan/kad/1.0.0 but i don’t think these are same.

udp connecton is ok as quic and webrtc already works in other applications. I also tried setting up multiple bootstrap server and adding each other to their bootstrap list but still the same.

now i can even do ipfs routing findpeer <a-device-that-is-behind-nat-without-open-port> from another device that has same condition and i get ok result(x.x.x.x is real peer public ip and y.y.y.y is my bootstrap public ip):

/ip4/x.x.x.x/udp/1025/webrtc-direct/...
/ip4/x.x.x.x/udp/1025/quic-v1/...
/ip4/y.y.y.y/tcp/4001/p2p/.../p2p-circuit
...

but when i do ipfs p2p forward... or ipfs cat <CID-of-some-file-that-only-exists-in-same-peer-as-above> i do get nothing(although ipfs routing findpeer worked fine for the peer)

4

Please run without docker, first at least. Kubo needs to be able to figure out the public facing IP and potentially punch hole through NATs. Once it works that way then you can see if you figure out docker rootless and any other shenanigans.

If a node is not publicly reachable, it will not announce to the public dht and content will not be discoverable unless you are connected already to it.

5

Hi @amirss,

a drive-by comment, noticed a few likely culprits.

1. The server profile

It is meant for datacenter machines with a static public IP. On a laptop behind a consumer router it silently disables several things you almost certainly want:

  • Discovery.MDNS.Enabled → false means no local peer discovery on the LAN.
  • Swarm.DisableNatPortMap → true means no UPnP / NAT-PMP, so your home router never opens a port to Kubo.
  • A long list of private / RFC1918 ranges in Swarm.AddrFilters and matching Addresses.NoAnnounce entries. On an isolated LAN this is the deal-breaker: your laptops are reachable only via private addresses, and the profile tells Kubo to neither dial nor announce them.

That fits your symptoms: AutoNAT marks the laptops private, only /ipfs/lan/kad/1.0.0 is joined, findpeer succeeds via your bootstrap, and the actual fetch stalls because the two laptops can never dial each other.

Suggestion: do not apply the server profile without reading the profile reference first; it bundles opinionated changes that are wrong for consumer networks. Set the individual keys by hand instead. On the laptops:

Add filters back only when you have a specific reason and specific ranges, and do it by hand. Blanket private-range blocking is the wrong default on a consumer LAN and painful to debug, since nothing looks broken yet nothing connects.

server is fine on the publicly reachable bootstrap node, where it actually fits.

2. Docker networking

Rootless Docker with pasta helps, but Kubo still sits behind an extra NAT layer between the container and the host. libp2p’s observed-address detection and DCUtR hole punching work much better when Kubo can see the real host interfaces.

For a clean test:

docker run --rm -it --net=host ipfs/kubo:v0.41.0

That drops the container NAT, lets Kubo bind to host interfaces directly, and lets UPnP / NAT-PMP reach your ISP router. Once port 4001 opens (automatic via UPnP, or by hand), AutoNAT should flip to public and /ipfs/kad/1.0.0 should appear in ipfs id. You can observe UPnP / NAT-PMP with GOLOG_LOG_LEVEL="error,nat=info"

3. Sanity checks

  • ipfs id lists /ipfs/kad/1.0.0 on any node you expect to be publicly reachable.
  • ipfs swarm peers grows beyond your bootstrap.
  • On the same LAN, mDNS alone should let the laptops find each other; the WAN DHT is not required for that.
  • If a laptop is stuck behind CGNAT / double-NAT, fall back to a circuit relay via your bootstrap. You already see /p2p-circuit paths in findpeer, so that side is wired up.

If ipfs cat still hangs, even with manual ipfs swarm connect share config, and we can keep digging.

Good luck.