How to speed up content discovery, hacks

TLDR, for hacks go to the end.

In the beginning of the year I’ve played with IPFS, the results were, that it wasn’t as useful IRL as it was promised. The main problem was that it was too slow for new or equally unpopular files compared to HTTP. I’m not talking here about seconds, but order of magnitudes slower. This really hampers RL adoption. I agree that it becomes fast once the file becomes more popular, but until it does it’s slow as hell and hence it comes to a viscious cycle: the file doesn’t become popular on IPFS, because HTTP is faster than IPFS, so then the availability on IPFS doesn’t increase and remains slow.

I’ve suggested improvements and others reported similar problems on this forum and on IRC.

I’ve went then to debug the issue more thoroughly and tried to reverse engineer/understand the protocol.

Sadly I’ve lacked time due to studies and I’ve lost a bit of motivation after I didn’t quickly got over the TLS handshake, but nevertheless I ocassionaly tried to experiment with IPFS or tried to use it in RL applications.

I think I’ve even found a bug/issue, see first hack.

HACKS

The interesting thing was that even when I’ve had two nodes connected together via VPN (so no issues in firewalls/NAT) it was slow. The trick to make it fast was to invoke dht findpeer <peerID> and then ping it through ipfs. If the pings stopped, quite soon after the download stalled.
So I’ve made a script that just repeats ipfs ping -n 1000 <peerID> all the time. That made the process almost as quick as HTTP.

The other interesting (and well known?) hack I’ve used to make it quickly available was to force download all the files through HTTP gateways (ipfs.io, cloudflare-ipfs.com, siderus.io). You could just write a CRON job to do this for you, go through a list of links once per day and wget the file through the gateway. Of course the first time was slow as hell, so it was best to set it on repeat as it was time-outing.

For HTTP backward compatibility you can just give HTTP links and “x-ipfs-path” HTTP headers where to find IPFS content, but sadly the browsers (which are compatible) don’t test what’s faster or rather have no idea what is faster.

Use IPFS links rather than IPNS. IPNS links are slow, they suffer the same issues as above. If the IPNS is not cached it’s super slow, it can be minutes.

1 Like

Depending on your VPN setup I think it’s likely that you may have accidentally been breaking mDNS — which would effectively break local peer/content discovery. Using a VPN also doesn’t necessarily remove any chance of firewall/NAT problems.

Multicast traffic doesn’t usually automatically span VPNs (maybe you’re using something that takes care of this?) or other network boundaries by default. You’d likely need to set up an mDNS server or something to repeat that traffic over the VPN.

yeah, IPFS keep boasting about what it can realize, but actually can’t.

My static site on IPFS is lightning fast (it’s not popular) if I have it pinned regularly and address it by CID. But as a matter of principle, I used IPNS for internal links instead of DNSlink, which makes it slow as hell most of the time on first load. After that it seems the IPNS lookup is cached locally (for a while) and it’s usually fast. But yeah, IPNS is the biggest problem.
https://js.ipfs.io/ipns/QmZJBQBXX98AuTcoR1HBGdbe5Gph74ZBWSgNemBcqPNv1W

1 Like