Does this means after upgrading go-ipfs? Or randomly? Actually, I see 0.4.22. You should try 0.4.23, alhtough probably the issue is not related.
This results in file retrieval operations from A <-> B failing
I am not sure this should affect file retrieval. There is a connection established between those two nodes.
Node A has a reference to Node B, but the IP address IS INCORRECT, referencing the internal k8s IP address…
I am a bit confused because Node B is supposed to be standalone, outside of kubernetes, so I’m not sure what the internal k8s IP address means here. By all means, the connection source address as seen by Node A was /ip4/192.168.15.104/tcp/10570.
Thanks for the replies! Hector - we had the same suspicion that it might not be related but noticed that the file retrieval issue is immediately resolved if we peer from inside k8s -> standalone (after explicitly disconnecting any localhost addresses, this ensures neither node A or node B have an internal k8s address)
I should have clarified, we are actually in the process of migrating from a set of standalone IPFS nodes outside k8s to a mix of IPFS k8s pods within k8s and standalone nodes outside k8s - prior to this migration we did not have this issue at all (file retrieval between nodes worked fine)
So, you are right that the connection seen by Node A is /ip4/192.168.15.104/tcp/10570 - however this is not the address that Node B actually lives at - trying to re-connect to that address from inside k8s ( /ip4/192.168.15.104/tcp/10570) simply does not work. I suspect the ingress routing of the request results in the final connection to Node A coming from a k8s local address.
When traversing the DHT to retrieve a file only present on Node B, it seems like Node A’s connection string for B is either invalid or retrieval fails at some other step…
If there is actually a live connection between the two (as swarms peers reported), then it should not need to traverse the DHT. There might be some weirdness in that setup, so that the connection seems to come from a different place and may not even work at all so that DHT fallback is necessary (maybe traffic only flows one way?).
In any case, if /ip4/192.168.15.104/tcp/10570 is what is seen rather than the real source IP outside k8s, I must only assume that k8s is doing some sort of address translation on ingress that is not really working (?). Unfortunately I am not so familiar to networking on kubertenes to say…
Yeah, this is exactly what we suspect is happening - that the connection ‘seems’ to come from a different place due to ingress address translation. This would not be an issue if file retrieval worked despite this odd local connection behavior but unfortunately it does not.