Kubo's lowpower profile impact on DHT announcements

Help IPFS
1

I’d like to better understand how peer lookup works and what the various parts of the IPFS pipeline are that could be slow.

I’m attempting to access a new CID that I recently pinned to my home Raspberry Pi 4 server, and I’m attempting to re-pin it on a different machine on a different network. These machines have talked to each other over IPFS before.

The CID is: bafybeih2r75jrkslgcs7b5orecbuypa5iohwnceeynr522ncwcbhi7nyr4

The PeerID of the Pi4 is /p2p/12D3KooWCFcfiBevjQD42aRAELMUZXAGScRiN2NcAthokF4dMnVU

The CID itself is fine, and IPFS check on fleek has no problems identifying it.
https://ipfs-check.on.fleek.co/?cid=bafybeih2r75jrkslgcs7b5orecbuypa5iohwnceeynr522ncwcbhi7nyr4&multiaddr=%2Fp2p%2F12D3KooWCFcfiBevjQD42aRAELMUZXAGScRiN2NcAthokF4dMnVU&timeoutSeconds=60

On the machine trying to access it ipfs swarm does not list the Pi4’s address, but it has gotten to Fetched/Processed 1337 nodes, so it started grabbing something, but now it’s stuck.

In the past I’ve had luck forcing the nodes to talk to each other by runnning:

ipfs swarm connect /p2p/12D3KooWCFcfiBevjQD42aRAELMUZXAGScRiN2NcAthokF4dMnVU

But I’d like to avoid that because the point of IPFS is that you don’t need to know who has the data a-priori, it should be able to find it.

I’d like to understand better what is causing the downloading node to get stuck, why it got some data, but then it hasn’t gotten the rest of it, and if I just let this wait (it’s been trying to grab data for over 20 hours now), if it will eventually get there.

2

It looks like your Pi4 is not announcing data on Amino DHT.

Your CID has no providers in DHT, it works only because you connected to other peer directly (try running the ipfs-check without specifying any Multiaddr):

With Direct Connection (no Peer lookup)

2024-09-20_17-41
2024-09-20_17-411046×830 114 KB

No Direct Connection (peer lookup in DHT)

2024-09-20_17-35_1
2024-09-20_17-35_1982×307 15.2 KB

If you lose connection, you lose the only provider of the data, and you wont be able to continue download until you connect to it again, manually, or by a a luck during random DHT walk.

Perhaps you’ve enabled the lowpower Kubo profile x on your raspberry Pi? I’ve seen people enabling it blindly on lowpower devices like rpi. This profile disables announcing content to the network. If this is the case, you can fix it by reseting Reprovider.Interval to 22h and restarting your daemon. It will start announcing things correctly.

3

It’s likely that I did use lowpower. Is there a way to tell given the config? Previously the Reprovider.Interval was set to “0s”. This is the updated version of my config: ipfs config pi4 - Pastebin.com

What is the main use-case of lowpower if it is not going to announce content? Is it mainly just reading from the network? If I start announcing content, do I lose advantages of lowpower? I’m also wondering why “22h”, could I set it to something more frequent?

If I run:

ipfs routing provide bafybeicajf2tn7bttkcdhhj4un7alfjbgx5hdhn6rsi66mnipwmrksen3y

Does that force the Pi to announce that it is providing the content, such that a downloading node will be able to pick it up more quickly? Or do I need to wait for the reprovider interval to elapse?

Is it correct that even after I announce I have a CID, other peers will forget about it after 1 day? So I need to constantly re-announce that I have everything I’m providing? If I ran that a chronjob with that routing script over all pinned CIDs, would it effectively be the same as letting the Reprovider.Interval ellapse?

EDIT: I’m noticing that when I run the routing provide with --verbose I get many messages that say things like:


12:30:56.474: error: failed to dial: failed to dial QmQGTz9cbgr83Yn4G8PeQkevibVkSi5SR8HF93JcAVdAEs: all dials failed
  * [/ip6/2a02:2f08:4d18:8b00:5054:ff:fec9:6153/tcp/4001] dial refused because of black hole
  * [/ip4/82.76.245.108/tcp/4001] dial tcp4 0.0.0.0:4001->82.76.245.108:4001: i/o timeout

12:30:56.642: error: failed to dial: failed to dial 12D3KooWGhW6f53sq82ifo32VKuer8ydR91GMGfcwGyBPxoypkcz: all dials failed
12:30:56.648: error: identify failed to complete: context canceled
12:30:56.648: error: failed to dial: context canceled

12:31:16.029: error: failed to negotiate protocol: stream reset

Wondering if these are issues or not.

4

Yes, as i’ve suspected you have announcements disabled.
Change it to 22h, restart the node, let ir tun for a day and you should be fine.

Yes, it is basically a read-only client, like IPFS gateway for your personal use that is not acting as a co-host of the data. Mostly legacy, these days we have better software for that use case (Rainbow).

I actually think setting it to 0s makes more harm than good and considering removing this from lowprofile so other people dont hit the problem you described.

Potential fix in feat: explicit announce-on|off profiles by lidel · Pull Request #10524 · ipfs/kubo · GitHub

DHT peers keep information for 48h since this PR, so you need to re-provide more often than that (that is why we have 22h as default).

Yes, but likely less efficient and more error prone. Just remove cron and use Reprovider.Interval=22h. If you only want to announce ONLY explicitly pinned data, you can set Reprovider.Strategy to pinned, but unless you have a lot of non-duplicated data, keeping it at all is fine and may be more efficient (because it does only a single pass over blockstore).

Amino DHT is a distributed network of random people running IPFS nodes. You can’t expect all them to be online all the time, or configured correctly, its normal to have failed dials while looking for online ones.

“/ip4/172.100.113.212/udp/4001/quic”,

replace legacy /quic listeners, it was superseded by /quic-v1

“Routing”: {
“AcceleratedDHTClient”: false,
“Methods”: null,
“Routers”: null,
“Type”: “dhtclient”
},

You may want to change Routing.Type to autoclient to ask both DHT and IPNI at cid.contact.