I deployed a private network, where there are multiple nodes connected to a single bootstrap node. For this example, let’s assume there are 3 nodes(A, B, C). A is the bootstrap node. B and C added A to their bootstrap nodes. When B adds a file, C can’t fetch that file until A fetches the file. Is it how it’s implemented, is there a way around or did I make any mistake?
Interesting, I don’t observe this behaviour! Here’s what I tried, maybe you’ll see something I’m doing differently (nodes are not behind NATs):
First I generate a key, copying it to Node A, B, and C:
ipfs-swarm-key-gen > ~/.ipfs/swarm.key
Then I create Node A, the bootstrap node:
ipfs config profile apply server
ipfs bootstrap rm --all
systemctl --user start ipfs
I grab an address I can bootstrap with too:
user@localhost:~$ ipfs id
{
"ID": "12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC",
"PublicKey": "CAESIIKtysHcZnikKrEY/OdafraHo1StpVSKbMQd6I06I++b",
"Addresses": [
"/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC",
"/ip4/139.177.196.92/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC",
"/ip6/2600:3c04::f03c:92ff:fe49:b939/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC",
"/ip6/::1/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC"
],
"AgentVersion": "go-ipfs/0.8.0/",
"ProtocolVersion": "ipfs/0.1.0",
"Protocols": [
"/floodsub/1.0.0",
"/ipfs/bitswap",
"/ipfs/bitswap/1.0.0",
"/ipfs/bitswap/1.1.0",
"/ipfs/bitswap/1.2.0",
"/ipfs/id/1.0.0",
"/ipfs/id/push/1.0.0",
"/ipfs/kad/1.0.0",
"/ipfs/lan/kad/1.0.0",
"/ipfs/ping/1.0.0",
"/libp2p/autonat/1.0.0",
"/libp2p/circuit/relay/0.1.0",
"/libp2p/fetch/0.0.1",
"/meshsub/1.0.0",
"/meshsub/1.1.0",
"/p2p/id/delta/1.0.0",
"/x/"
]
}
I’m using /ip4/139.177.196.92/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC in this example.
I create Nodes B and C the same way:
ipfs init
ipfs config profile apply server
ipfs bootstrap rm --all
ipfs bootstrap add /ip4/139.177.196.92/tcp/4001/p2p/12D3KooWJcUsXsgHiiwSz1RCtB8Y1LzGUhfn4rYoSRrR3Gta9cbC
systemctl --user start ipfs
Then I test:
First I verified each node could see each-other with ipfs swarm peers, I had 2 lines of output on each node, signifying each node is aware of the other 2.
I created a file named testfile on Node B:
Hello world :)
Node B
I add the file to Node B.
user@localhost:~$ ipfs add testfile
added QmTS1nviAatBXrpSDdvXtRbnne8FaT8n11ZpZ6Hx7FEPWF testfile
15 B / 15 B [===========================================] 100.00%
Node C
I can retrieve it on C instantaneously 
.
user@localhost:~$ ipfs get QmTS1nviAatBXrpSDdvXtRbnne8FaT8n11ZpZ6Hx7FEPWF
Saving file(s) to QmTS1nviAatBXrpSDdvXtRbnne8FaT8n11ZpZ6Hx7FEPWF
15 B / 15 B [======================] 100.00% 0s
user@localhost:~$ cat QmTS1nviAatBXrpSDdvXtRbnne8FaT8n11ZpZ6Hx7FEPWF
Hello world :)
I didn’t change the profile to server. Does that effect the expected behaviour? 
I’ll try changing the profile to server and see.
1 Like
Nope. Some times it works but most of the time it doesn’t. I waited for around 20-30 min but still no use.
What’s your output of ipfs swarm peers? Do the nodes just not see eachother?
If you see some nodes but not others, is there a pattern there? (Behind NAT)
Are you able to try with IPv6?
Edit: BTW is this you? distributed system - can't fetch files from other nodes - ipfs - Stack Overflow
In bootstrap node I’m getting all the other nodes but in other nodes I can only see Bootstrap nodes. I have 2 EC2 instances, where I changed Security Policy to allow all the traffic. Except for these 2 I think all other nodes are behind NAT.
If this is a NAT issue, have you considered using Relaying to defeat it? You could make the NATless bootstrap nodes into relays for your private network. Using AutoRelay I believe nodes should be able to discover the advertised relays.
Also FWIW you can try manually connecting to nodes using ipfs swarm connect, this could help with diagnosing.
Thanks for the help. Relay doesn’t seem to work for me. Though ipfs swarm connect is working I don’t think it’s a solution for the problem as it is impossible to add every node in other nodes.
Sorry for the delay!
If you’re able to manually connect two nodes behind the NAT, then the bootstrap node should be advertising those peers to eachother, allowing them to all connect to each-other and send data. The reason I suggested ipfs swarm connect is to assist in diagnostics.
What should be happening is you connect to a bootstrap node, that bootstrap node advertises it has us as a peer when other nodes connect. When we add a file, assuming you’re using a DHT, that file should get added to the DHT, and then the bootstrap node will be able to let other nodes know they can get the file from your node. Assuming they’re able to connect (ipfs swarm connect), which is supposed to happen transparently, automatically, they’ll connect to the node, and request the file.
If they’re unable to connect though, that’s where relaying comes in. Either using AutoRelay, or another mechanism to advertise the relay pathway. For a demo we’ll be publishing in the near future, I use a python script on a relay node, and advertise the relay pathway of connected clients over PubSub, which allows clients to discover each-other over the relay if they’re stuck behind NAT (I use that because the demo is browser-based, there should be better solutions for go-ipfs…).
I hope this helps, if you need more specific info on relaying, please don’t hesitate to ask. I know figuring some of this out can be daunting.
Thanks a lot. It’s a side project, which I started to explore new technologies. I work on it, in my free time, which is a bit scarce this days 
. I’ll look into relaying. I’ll post a comment if I get struck.
1 Like
I’m facing the same problem as @abhijeet_ar .
My private network topology is
NodeA <—>NodeB(bootstrap) <—> NodeC.
NodeA is a peer of NodeB and NodeB is a peer of NodeC. If i try to upload a file to NodeC and try to fetch the file from NodeA(indirect peer) it returns empty but if i try to fetch the file from NodeB (direct peer) it works.
Does IPFS not support indirect propagation of file? This seems to be a pretty basic feature
The issue is mentioned in detail over here.