Manual connection between node A and B works but

The only way to transfer files between my nodes (say: A and B) is to run ipfs swarm connect on both nodes.
Both 4001 and 5001 ports (on nodes A and B) are open as well as NAT on my modem.
Given that I don’t want to enable UPnP, am I miss something in network configuration?
Thanks so much.