How to construct proper IPNS-Over-Pubsub fetch

Help
, ,
1

Hello everyone, glad to be here. We’re trying to set up IPNS and IPNS-Over-Pubsub in helia. The default behavior of IPNS-Over-Pubsub router in helia is to wait to receive the the IPNS update from its gossipsub subscription. But that takes too long, and in Kubo there’s a way for asking a connected peer directly for the latest IPNS record. The difference in terms of latency is huge, helia takes ~30s to resovle an IPNS, while kubo ~2s.

We would like the helia node to fetch the latest IPNS record immediately instead of waiting for IPNS updates in the pubsub topic. The IPNS-Over-Pubsub protocol suggests using Libp2p-Fetch for this, which is what Kubo is using. But we’re hitting a roadblock because no matter how we construct the fetch identifier or key, kubo nodes always respond with NOT_FOUND

This is how we’re constructing fetch key at the moment. Please let me know how to correct it:

const subplebbitIpnsName = "12D3KooWJ7mvJFaWHK43MYd1Au4W4mkbY7L8dQaiMBqH5bZkSsFn";
const subplebbitIpnsAsPeerId = PeerId.parse(subplebbitIpnsName);
const fetchKey = "/ipns/" + uint8ArrayToString(subplebbitIpnsAsPeerId.toBytes(), "binary");
const res = await helia.libp2p.services.fetch.fetch(peer.id, fetchKey); // always gives undefined because kubo responds with NOT_FOUND

Moreover, is implementing libp2p-fetch for IPNS on the roadmap for helia?

References:
IPNS Pubsub router: IPNS PubSub Router

1 Like
2

This is something of an edge case.

Given that the identifier type in the fetch spec is string, you’d think you could use the IPNS Name string representation to get the record value.

As it happens, go-libp2p-pubsub-router’s fetch implementation accepts datastore keys as identifier - internally the go implementation of IPNS stores records in the datastore with the key "/ipns/" + peer-id-multihash-bytes, so that’s what we need to send.

This is easy to do in golang because it can cast strings to byte[] and back again. JavaScript is not so lucky as it’s strings are UTF-16, so round tripping Uint8Arrays usually results in garbage when byte sequences can be interpreted as either multi-byte or non-printable characters.

I’ve opened fetch: Change identifier pb type from `string` to `bytes`? · Issue #656 · libp2p/specs · GitHub to update the spec to allow sending identifier as bytes instead of string, and also fix!: accept Uint8Arrays as keys by achingbrain · Pull Request #2909 · libp2p/js-libp2p · GitHub to update the JS implementation to accept Uint8Array fetch identifiers.

When it’s published you should be able to construct the correct fetch key with:

import { peerIdFromString } from '@libp2p/peer-id'
import { concat } from 'uint8arrays/concat'
import { fromString } from 'uint8arrays/from-string'

const subplebbitIpnsName = "12D3KooWJ7mvJFaWHK43MYd1Au4W4mkbY7L8dQaiMBqH5bZkSsFn";
const subplebbitIpnsAsPeerId = peerIdFromString(subplebbitIpnsName);
const fetchKey = concat([
  fromString("/ipns/"),
  subplebbitIpnsAsPeerId.toMultihash().bytes
]);

const res = await helia.libp2p.services.fetch.fetch(peer.id, fetchKey);

Moreover, is implementing libp2p-fetch for IPNS on the roadmap for helia?

This should be simple to add to @helia/ipns once the underlying support for Uint8Array keys is in @libp2p/fetch.

3 Likes
3

Thanks for the help with this. I just tried v3.0.0 of @libp2p/fetch and it worked.

Can we expect @libp2p/fetch to be incorporated within @helia/ipns soon? I made this github issue about it