IPNS with known peers

I have a use case where peers should be able to share with each other a number of mutable pin sets–so probably IPNS records pointing to a file containing a list of CIDs, or a folder or something.
Readability is not a concern, but patience is.

I noticed that it’s possible to ipfs swarm connect to a specific peer.
Does connecting to a specific peer mean that the IPNS records they publish can be resolved quickly?
Are there hidden disadvantages to this approach?