Can't access published IPNS record, config problem?

iohzrd · May 23, 2019, 3:32am

First, a little bit of background so y’all know where I’m coming from…

I started a project a couple months back. The goal was to make a completely decentralized publishing / subscription platform that could replace YouTube and social media platforms in general. In other words, it enables the publication of arbitrary content and subscription to the publishers in a censorship resistant way…

The original architecture is as follows: Use a DHT to serve in response to a requests for a particular public key, a magnet link which points to a JSON file containing profile information, including other magnet links which represent a users publications. This magnet link would be downloaded using webtorrent and the JSON would be passed into a UI(electron+angular) to be rendered appropriately. As a “feed”.

The key here is the DHT responds to a request for a persistent identity(pubkey) with data that is mutable/updatable(only by the holder of the corresponding private key).

I played around with a couple DHT libraries but after experimenting with IFPS/IPNS a bit I decided to make the switch. Now, instead of the DHT responding with a magnet link, I’m using IPNS to respond with an IPFS link which points to the aforementioned, latest json file.

To be clear, the goal of this is NOT for profiles/content to be accessible via a regular browser / http/s. Rather, to have electron acquire them directly through ipfs, which would already work if IPNS was more reliable / i could figure out how to configure it properly.

the abstract flow goes something like this…
IPNS / dht.GET(pubkey) -> ipfs / magnet link -> profile.json(list of ipfs /magnet link(s) to users publications) -> magnet/ipfs link -> content -> render in electron(already working, albeit very beta)…

Development has been going well, but I just can’t seem to reliably get IPNS to serve the IPFS links to the identity file. I’ve tried customizing the ipfs config but to no avail…

for reference: https://www.github.com/iohzrd/follow

Please let me know if you’d like further clarification or if you have any suggestions or thoughts…

postables · May 23, 2019, 4:05am

Probably would make debugging easier if you could post links to source code responsible for performing these operations. That being said, IPNS can be pretty slow at the moment, but there’s a few things you can do to enable better performance, and discovery of records.

For better record publishing discovery, you can using the daemon flag --enable-namesys-pubsub however for this to properly work, other nodes will need to subscribe to the same topics of your records.

Additionally one thing to keep in mind with the availability of your records, is that they have lifetimes associated with them. Normally if the daemon or service being used to publish the reccord is always online, this isn’t a problem. However, if the node or service that was responsible for publishing the records does actually go offline, then the record won’t be republished and can dissapear from the network. A way to mitigate this is to tweak your lifetimes, and optionally your ttl.

iohzrd · May 23, 2019, 4:15am

github.com

iohzrd/follow/blob/master/main.js

/* eslint-disable space-before-function-paren */
'use strict'
const fs = require('fs')
const path = require('path')

let WebTorrent
try {
  console.log('loading webtorrent-hybrid...')
  WebTorrent = require('webtorrent-hybrid')
} catch (error) {
  console.error(error)
  console.log('failed to import webtorrent-hybrid, falling back to webtorrent')
  WebTorrent = require('webtorrent')
}
const client = new WebTorrent()

// Import Electron dependencies
const {
  BrowserWindow,
  Menu,

This file has been truncated. show original

github.com

iohzrd/follow/blob/master/ipfs-config.js

'use strict'
const os = require('os')
const IPFSRepo = require('ipfs-repo')
const path = require('path')
const wrtc = require('wrtc')
const WStar = require('libp2p-webrtc-star')
const wstar = new WStar({ wrtc })

const repoPath = '/home/iohzrd/.config/follow/ipfs'
// const repoPath = path.join(os.homedir(), '.follow')
const repo = new IPFSRepo(repoPath)

module.exports = () => ({
  repo: repo
  // config: {
  //   Addresses: {
  //     Swarm: [
  //       '/ip4/0.0.0.0/tcp/4001',
  //       '/ip6/::/tcp/4001',
  //       '/ip4/0.0.0.0/tcp/4002',

This file has been truncated. show original

iohzrd · May 23, 2019, 4:45am

I’m not running IPFS as a daemon, rather programmatically via node/electron. Any idea what the config equivalent of that flag would be? Also, I’m aware of the record lifespan. That’s an issue I plan to address once i get there.

postables · May 23, 2019, 4:58am

I don’t think there is a config equivalent of that flag, however when initializing your IPFS via the require statement, I believe you can pass in options like so.

When you say I just can't seem to reliably get IPNS to serve the IPFS links to the identity file what are the specific issues you’re experiencing?

iohzrd · May 23, 2019, 5:12am

Sorry for the ambiguity. I think my usage of the term publish and subscribe might be getting confused with the pubsub system, which I am not using, and don’t believe i need to. I’m simply saying that I can’t retrieve IPNS records(that i published, from another computer) , like at all. but i’ll give it a shot anyway.

postables · May 23, 2019, 5:16am

No problem, namesys pubsub is just a method of assisting the distribution and discovery of IPNS records, so it can potentially help alleviate being unable to discover IPNS records if configured correctly.

The two computers you’re using, the one publishing the record and the one are they on the same network, or different networks? Additionally, depending on the setup of your IPFS nodes they could be either running into issues with firewalls, or NAT and thus unable to either properly talk to other nodes, interact with the DHT, etc…

iohzrd · May 23, 2019, 5:19am

The two computers I’ve been testing with so far are both on the same local network, though i have a couple friends who will be helping me test soon. Testing the suggestion now…

iohzrd · May 23, 2019, 5:30am

Same result so far, with ipnsPubsub enabled or disabled, I get the following error when trying to retrieve the content associated this identity: QmcqwzjrpjRwrGFvUHptSpXTST3aWRrAyxohAfHfScQ172

the identity after the /ipns/ prefix seems like it’s being interpreted/formetted incorrectly. Maybe an error in ipfs itself?

the file it is supposed to be pointing to is accessable:
https://gateway.ipfs.io/ipfs/QmfPftrMH1EFSKQqjLoD8bKHCYhXxLLExCfn65PwHCtukZ

iohzrd · May 23, 2019, 7:07am

I know the way I’m doing it should work because I have a test file that works great, but only when resolving the ipns link of itself.

ipfs.on('ready', () => {
  ipfs.id((err, id) => {
    if (err) {
      return console.log(err)
    }
    console.log('ipfs.on ready')
    console.log(id.id)
    ipfs.name.resolve(id.id, function (err, name) {
      if (err) { throw err }
      console.log('resolve')
      console.log(name)
      ipfs.get(name.path, function (err, files) {
        console.log('get')
        if (err) { throw err }
        files.forEach((file) => {
          console.log(file)
          console.log(file.path)
          console.log(file.content.toString('utf8'))
        })
      })
    })
  })
})

swarleynunez · May 28, 2019, 7:47pm

I have the same problem. Have you found the solution?

iohzrd · June 1, 2019, 3:13am

No, I still cant access ANY IPNS records outside of from the node that published them… really wish I could figure out how… Default config or otherwise.

DougAnderson444 · June 24, 2020, 3:33am

@postables how do you get these records?

DougAnderson444 · June 24, 2020, 4:08am

I can’t resolve my IPNS links outside the publishing node either. I think I’ve tried everything.

I even have my js-ipfs node in the browser websocket connected to my go-ipfs server node. I thought that would encourage the dht to work faster. No luck there. Preload is enabled. Doesn’t seem to matter.

Do both nodes need to be running the same language (js/go)?

I even have EXPERIMENTAL: { ipnsPubsub: true } enabled in the js-ipfs node publishing it, and --enable-namesys-pubsub enabled in the go-ipfs node trying to resolve it. No luck with pubsub. As far as I can tell, before pubsub is any use the dht needs to work for the first time around, and that just never seems to happen between my two nodes.

I can’t figure out why it’s not working. The only other thing I can think of is just use ipfs-http-client and push it to the go-ipfs on the server and hope it eventually replicates through the network.

Or hack it a bit and try to figure out how to pubsub sub the ipns record.

Open to any suggestions…

DougAnderson444 · June 24, 2020, 5:34am

Ok, I think I’ve isolated the problem.

If you ipfs.name.publish(path, {key: 'self'}) then it resolves quickly.
If you ipfs.name.publish(path, {key: 'myCustomeKey'}) then it never seems to resolve on another node.

I suspect that peer-id key (‘self’) gets put on the DHT, and the non-self keys do not, which is why we are not seeing the other keys propagate? But that’s just a guess…

vasco-santos · June 25, 2020, 7:59am

Hey @DougAnderson444

js-ipfs does not have the DHT enabled by default, since we need to stabilize it. We will be working on getting a SPEC withthe latests DHT changes in go-ipfs so that we can update it in js.

With the above into consideration, IPNS in js land needs IPNS over pubsub enabled. You might be experiencing issues with the propagation delays through the network.
Can you try the following flow?

name.resolve() , which subscribes the topic
wait to guarantee the subscription
wait for the other peer to get notified of the subscription
publish new ipns record
wait until the record is received
resolve ipns record

Example of a test using this flow between go and js: https://github.com/ipfs/interop/blob/master/test/ipns-pubsub.js#L89-L115

From what I remember from IPNS, I don’t think the key should be the problem. Perhaps, with myCustomKey we need to create the key which makes things slower. Could you open a PR with a failing test with the key on https://github.com/ipfs/js-ipfs/blob/master/packages/ipfs/test/core/name-pubsub.js ?

DougAnderson444 · June 28, 2020, 11:23pm

Thank you @vasco-santos an @adin for pointing me in this direction, it’s of great help.

TL;DR:
What I am seeing is: with the js-ipfs node running in the browser, when I do go-ipfs name resolve <PeerId> in the CLI on the server, or through ipfs-http-client, Go isn’t resolving the PeerId, and the PeerId isn’t being added to either pubsub list.

Details:

My experiment results follows:
js-ipfs 0.47
go-ipfs 0.6.0

To reproduce:

spin up js-ipfs in browser & get the peerId
in go-ipfs, do ipfs name resolve <peerId>
(followed by Error: could not resolve name)
in go-ipfs, do ipfs pubsub ls and ipfs name pubsub subs
peerId is not added to pubsub list after resolve fails?

I ran the interop tests, and they all pass. But I am observing different results in production compared to the tests.

In the test, we get:

nodeGo.name.resolve
nodeGo.pubsub.ls() shows /record/PeerId(b64)
nodeGO.name.pubsub.subs() shows /ipns/PeerId

But in production, with
A. Websocket between JS and Go to ensure swarm is connected
B. swarm.peers() shows they are connected
C. ipnsPubsub, --enable-pubsub-experiment --enable-namesys-pubsub enabled

When we do ipfs name resolve in the CLI (or through http-client),

Error: could not resolve name
ipfs pubsub ls does not show /record/PeerId(b64)
ipfs name pubsub subs does not show in /ipns/PeerId

Bottom line:
Pubsub is working, but I wonder if there’s something missing in the pubsub side (both name and regular) in Go when the name doesn’t initially resolve. I still can’t figure out why Go doesn’t resolve a PeerId when they are clearly connected and pubsub is enable everywhere. I’m a bit stumped.

DougAnderson444 · June 29, 2020, 12:02am

@adin Just wanted to add: does/should go-ipfs add to the name pubsub subs list even if the resolve fails? I wonder this is where it’s getting hung up?

iohzrd · July 12, 2020, 7:00pm

i ended up just using the http interface with the go binary.
The project is very beta but it is working…

DougAnderson444 · July 14, 2020, 12:01am

I finally got this working after studying all this advice and many many experiments. My observations are:

A. In the js-ipfs, just like in the tests in ipfs.name.resolve it MUST include ( { stream: false } ) and ipfs.pubsub.publish MUST include ( { resolve: false } ), otherwise looks like the resolves returns empty or as the /ipfs/Qmhash of the go-ipfs node, which can be very confusing indeed.

B. Unlike the tests, I didn’t need to await the initial resolve on the server before continuing with publish and resolve. In fact, I think the await was causing my websocket to disconnect while waiting for the promise to resolve, thus causing issues with pubsub.

I’ve created a draft PR with my working solution example if anyone else wants to take it for a spin.

Thanks @vasco-santos and @adin!!