Swarm peers not discovering electron, orbit -> node env

Help
1

Hi

In my case I have ipfs nodes in the node environment and I can’t get them to connect, i can’t see the hash id of the other nodes in my peers, how can i discover them? it’s supposed to be able to get discovered in node environment right without use of w*-star?..

setInterval(async()=>{
			let peers  = await ipfs.swarm.peers()
			peers.forEach((e)=>{
				console.log(e.peer)
			})
		}, 5000)

I would suppose that by bootstrapping I should find my nodes on the network?
DHT random walk, pubsub?

relay: {
				enabled: true,
				hop: {
					enabled: true,
					active: true
				}
			},
			pubsub: {
				enabled: true,
				emitSelf: true
			},
			dht: {
				kBucketSize: 20,
				enabled: true,
				randomWalk: {
					enabled: true,
					interval: 300e3,
					timeout: 10e3
				}
			}

Default Bootstrap

[
    '/ip4/104.236.176.52/tcp/4001/p2p/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z',
    '/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ',
    '/ip4/104.236.179.241/tcp/4001/p2p/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM',
    '/ip4/162.243.248.213/tcp/4001/p2p/QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm',
    '/ip4/128.199.219.111/tcp/4001/p2p/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu',
    '/ip4/104.236.76.40/tcp/4001/p2p/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64',
    '/ip4/178.62.158.247/tcp/4001/p2p/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd',
    '/ip4/178.62.61.185/tcp/4001/p2p/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3',
    '/ip4/104.236.151.122/tcp/4001/p2p/QmSoLju6m7xTh3DuokvT3886QRYqxAzb1kShaanJgW36yx',
    '/ip6/2604:a880:1:20::1f9:9001/tcp/4001/p2p/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z',
    '/ip6/2604:a880:1:20::203:d001/tcp/4001/p2p/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM',
    '/ip6/2604:a880:0:1010::23:d001/tcp/4001/p2p/QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm',
    '/ip6/2400:6180:0:d0::151:6001/tcp/4001/p2p/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu',
    '/ip6/2604:a880:800:10::4a:5001/tcp/4001/p2p/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64',
    '/ip6/2a03:b0c0:0:1010::23:1001/tcp/4001/p2p/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd',
    '/ip6/2a03:b0c0:1:d0::e7:1/tcp/4001/p2p/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3',
    '/ip6/2604:a880:1:20::1d9:6001/tcp/4001/p2p/QmSoLju6m7xTh3DuokvT3886QRYqxAzb1kShaanJgW36yx',
    '/dns4/node0.preload.ipfs.io/tcp/443/wss/p2p/QmZMxNdpMkewiVZLMRxaNxUeZpDUb34pWjZ1kZvsd16Zic',
    '/dns4/node1.preload.ipfs.io/tcp/443/wss/p2p/Qmbut9Ywz9YEDrz8ySBSgWyJk41Uvm2QJPhwDJzJyGFsD6',
    '/dns4/node2.preload.ipfs.io/tcp/443/wss/p2p/QmV7gnbW5VTcJ3oyM2Xk1rdFBJ3kTkvxc87UFGsun29STS',
    '/dns4/node3.preload.ipfs.io/tcp/443/wss/p2p/QmY7JB6MQXhxHvq7dBDh4HpbH29v4yE9JRadAVpndvzySN'
  ]
2

Hello @zorrillosdev

Are all these peers running in the same network? If so, they should be discovered via https://github.com/libp2p/js-libp2p-mdns which is enabled by default in js-ipfs configuration for node.

If the nodes are a distinct network, they will not be discovered out of the box currently. You can see here what peer discovery modules are currently supported in JS: https://github.com/libp2p/js-libp2p/blob/master/doc/CONFIGURATION.md#peer-discovery

The DHT random walk would be the ideal scenario. However, we do not have it yet enabled by default in JS: go-ipfs made huge improvements in the GO implementation of the DHT and we plan to bring it to JS in a few months. We should get a SPEC soon to start its implementation. I am also working on https://github.com/libp2p/js-libp2p-rendezvous/pull/6 that will also be a good solution for discovery. Meanwhile, if you are out of the same network, you can try enabling the DHT in the IPFS config, or use webrtc-star. As I see for your config, you are trying out the DHT, but no luck discovering the peers.

Can you try to ping the peers from your node to check if they are reachable? There could be a network barrier between them

3

Hi @vasco-santos thanks for reply.

I get some solution in progress and here is my answers:

  1. Peers are not on the same network
  2. I am currently using webrtc with signaling server to discover peers.
  3. Good to know. JS DHT coming soon…
  4. Actually i can ping the peers and currently they can get connected using wrtc-start

Now my scenario is as follows:

  1. Node B (js-ipfs) connected with Node A (go-ipfs) in same network with swarm.peer.connect(Node A) so far so good.
  2. Node C,D,F (js-ipfs) discovered each others with Node B (js-ipfs) using webrtc-star so far so good too.

The question is:
How can i proxy the peer discovering to Node A using Node B to connect Node C,D,F to Node A?..

I saw an example that you implemented very similar to what I am trying to do, the only thing that I am not able to do is connect to Node A from C, D, F (out of network) using node B as kind of bridge…

Based on the diagram below, my nodes would be in my case the nodes B, C, D, F that are connected using webrtc. Node B connects directly with Node A using ipfs.swarm.connect(/ipv4/local_ip/tcp/4002/ws/p2p/NodeAHash) but C,D,F are never known with Node A despite being connected with Node B.

Notes:

  • Node A would be the desktop terminal equivalent of the diagram below…
  • B,C,D,F nodes are running in node environment…

Captura de Pantalla 2020-07-10 a la(s) 21.43.56
Captura de Pantalla 2020-07-10 a la(s) 21.43.56942×438 9.52 KB

My theories and possible solutions to consult you:

  • Set Node B with loopback ip address ipfs.bootstrap.add(/ipv4/127.0.0.1/tcp/4002/ws/p2p/NodeBHash) to serve as bootstrap. Could this approach help nodes C, D, F to find A using B like some kind of pipe?
  • Let MDNS do the job?
  • Need read more XD? and where can i read more how implement this solution?

I really thank you for the help

4

Thanks for the detailed explanations.

The go-ipfs node will not be able to discover the other peers, but should be able to dial them. We are working on better protocols for discovery in this cases (like rendezvous https://github.com/libp2p/specs/tree/master/rendezvous or peer-exchange protocol).

Shortly, mdns is not supported in the browser and those browser nodes are only announcing themselves via the signal server. However, both browser nodes and go-ipfs nodes should be able to dial the other ones through the common known node. For this, they should use this node as a circuit relay. You can check out how to configure it on https://github.com/ipfs/js-ipfs/tree/master/examples/circuit-relaying

The missing piece is the discovery. At first, I recommend that you get a setup with the common known node as a circuit relay and try to dial between each other through it (manually). Once you have this working, you can get the discovery in a “hacky” way, while we don’t have a protocol ready for this purpose. My suggestion would be nodes subscribing a topic and when they discover each other, they publish in that pubsub topic the multiaddr of the peer they discovered. Then, the subscribers will receive the multiaddr of the peer (this multiaddr should include the circuit relay addr)

Hope this helps you in a next step. Let me know :slight_smile:

5

Hello @vasco-santos

A) I achieved something by establishing a go-ipfs node with my pinned data and using it as bootrstrap of my js-ipfs application, so I have managed to replicate my data between the js-ipfs apps, the strange thing is that now if I want to use Node A (go ipfs) and expand my network with another Node B (go ipfs) with the same data, according to the white paper (3.4 Block Exchange - BitSwap Protocol) Node A would try to find the data block in Node B of not having it and vice versa? Node A is connected to Node B directly. Should js-ipfs be able to get the data from both nodes? Being that Node A would try to get it from Node B or vice versa even though js-ipfs does not have a direct connection with Node B. I quote verbatim "In some cases, nodes must work for
their blocks. In the case that a node has nothing that its
peers want (or nothing at all), it seeks the pieces its peers
want, with lower priority than what the node wants itself "

B) If i have my js-ipfs node env empty and start replicate data from a list of multihash which is better if i want serve bitswap blocks ipfs.cat or ipfs.get? Which of these options leaves a copy of the replicated data at my premises? (no pinned needed) according this https://docs.ipfs.io/reference/cli/#ipfs-get looks like get saves data in disk… then i can exchange blocks via ipfs-js to other js-ipfs peers rigth? because the other peers can find the data in the js-ipfs nodes, and don’t need to request only to Node A for data?

6

btw sorry for all my confusing questions!! XD and thanks for reply

7

Happy to help :slight_smile:

a) If I understand correctly, you will have Node A (go-ipfs) and Node B (go-ipfs) for expanding your network) connected to each other. A Node C (js-ipfs) will have a connection with Node A.
Can Node C fetch content that is only available node Node B? This is a good question.
It can, but I think it cannot do it out of the box because of Node C (js-ipfs) not having the DHT for doing DHT queries. As a result, you will need to configure Node C with a delegate router for Node A, so that it can leverage Node A DHT to find content over the network (even for a Node D, that is only connected to Node B). I have this PR with an example on how to configure such setup using a public router: https://github.com/libp2p/js-libp2p/pull/507
You will want to use NodeA as the delegate instead of the public one (the public should work though if you are connected to the public network, but with bigger queries across the network). For setting up your own and using Node C (js-ipfs in a browser environment), you will need to configure a reverse proxy for Node A’s API with SS. Otherwise, js-ipfs node in the browser will be blocked by CORS when trying to leverage Node A’s API.

b) can you open a new forum discussion for that? I am not entirely sure, and I think someone else will have better answers for you regarding the files API/bitswap.