Only peers found from DHT seem to be getting used as relays, so can't use HTTP routers

Help Kubo
1

There also doesn’t seem to be any way/API to add peers to the relay list, and for our use case we only use HTTP routers, so we have no way for any of our discovered peers to ever be used as relays.

2

what type of HTTP router? Relays are peers that run a libp2p service used specifically to coordinate NAT hole punching and nothing beyond that.

3

our app is https://seedit.app it’s a reddit alternative but fully P2P using IPFS, IPNS over pubsub and gossipsub.

we bundle kubo with our electron application to distribute it, and when DHT was enabled, our users would often complain that it would kill their router (this still happens even recently). We also run gateways for the web version and DHT would often cause the VPS to become unresponsive, I guess because something on the network crashes because of DHT and it had to be manually restarted from the VPS dashboard. So we decided to create “IPFS trackers” (like bittorrent trackers). We have a few volunteers that run a key value store server and store peers, and it tries to implement the delegated routing API, and the kubo PUT API, and hopefully eventually the delegated routing POST API will be released officially and we can use it. this is an example of our kubo config:

"Routing": {
		"Methods": {
			"find-peers": {
				"RouterName": "HttpRouterNotSupported"
			},
			"find-providers": {
				"RouterName": "HttpRoutersParallel"
			},
			"get-ipns": {
				"RouterName": "HttpRouterNotSupported"
			},
			"provide": {
				"RouterName": "HttpRoutersParallel"
			},
			"put-ipns": {
				"RouterName": "HttpRouterNotSupported"
			}
		},
		"Routers": {
			"HttpRouter1": {
				"Parameters": {
					"Endpoint": "https://routing.lol"
				},
				"Type": "http"
			},
			"HttpRouter2": {
				"Parameters": {
					"Endpoint": "https://peers.pleb.bot"
				},
				"Type": "http"
			},
			"HttpRouter3": {
				"Parameters": {
					"Endpoint": "https://peers.plebpubsub.xyz"
				},
				"Type": "http"
			},
			"HttpRouter4": {
				"Parameters": {
					"Endpoint": "https://peers.forumindex.com"
				},
				"Type": "http"
			},
			"HttpRouterNotSupported": {
				"Parameters": {
					"Endpoint": "http://kubonotsupported"
				},
				"Type": "http"
			},
			"HttpRoutersParallel": {
				"Parameters": {
					"Routers": [
						{
							"IgnoreErrors": true,
							"RouterName": "HttpRouter1",
							"Timeout": "10s"
						},
						{
							"IgnoreErrors": true,
							"RouterName": "HttpRouter2",
							"Timeout": "10s"
						},
						{
							"IgnoreErrors": true,
							"RouterName": "HttpRouter3",
							"Timeout": "10s"
						},
						{
							"IgnoreErrors": true,
							"RouterName": "HttpRouter4",
							"Timeout": "10s"
						}
					]
				},
				"Type": "parallel"
			}
		},
		"Type": "custom"
	},

Note that because the kubo HTTP routing has a bug and doesn’t announce the correct addresses, we usually use a localhost reverse proxy that rewrites the addresses to be what ipfs id outputs.

So as you can see, we don’t use the DHT at all, but our users do discover many peers and connect to many peers, but none of these peers can be added as relays, because there’s no API for it, and because the relay system seems to only consider peers discovered from the DHT.

A scenario where a user might want to use a relay:

  • they are behind a NAT
  • they want to publish some content
  • they can announce to the trackers, but they don’t have any listen address that are reachable
  • so ideally they would pick one of the peer they know about (and maybe are even already connected to), and if they support relaying, they would use them as a /p2p-circuit listen address to do hole punching
  • this will become an even bigger issue once kubo supports webrtc using relays as signaling servers, as we won’t be able to use it at all
1 Like
4

Disable QUIC:

$ “Transports”: {
“Multiplexers”: {},
“Network”: {
“QUIC”: false,
“WebTransport”: false,
“WebRTCDirect”: false,
“TCP”: true,
“Websocket”: true
},

Tune down LIBP2P_SWARM_FD_LIMIT=25 (or more, default 160), reduce ConnMgr.GracePeriod to 5s. AcceledatedDHTClient needs 6000 thousands connections every hour, but 0 connections for lookups in between, so depends if routers can handle it (can’t say). Consider increasing ConnMgr High/LowWater defaults to avoid churn (400/100 if routers can handle it) ?

Unless it is running out of memory, I am not sure what would be going on.

What addresses does it announce and where?

I’m not sure about the libp2p specifics here, but are they configured as relays so that they’re exposing the libp2p protocols to act as such? In principle, I don’t see how or why things would be wired up to DHT-peers only, as far as libp2p go the DHT is just a routing layer implementation.

5

we already tried several methods discussed in the kubo github issues in the last few years and nothing really worked so we have abandoned using the DHT for now. we do plan on using DHT as a fallback (similar to how bittorrent has trackers and DHT as a fallback) but we are focusing on other aspects of improving the app at the moment. Kubo and libp2p-js/helia both work perfectly without the DHT and are really lightweight so it is better for us this way at the moment.

I don’t either and I don’t have the skills to debug it. The issue stopped when we stopped using the DHT.

It announces what is set in Addresses.Swarm instead of the actual listen addresses We advertise our listen addresses to the http router · Issue #10087 · ipfs/kubo · GitHub

we announce our peers to “trackers” servers that we run that implement the delegated routing api example: https://routing.lol/routing/v1/providers/bafkreic2vguwwzo4dddbxjas4pzlpbujxxi7erqfkwhmm2pls6c4q6iizm

we don’t “delegate” using the DHT though, it’s just a key value store server

all kubo nodes are configured by default as relays if they are publicly reachable, so yes, we don’t modify these configs. we know our peers work as relays because using Swarm.RelayClient.StaticRelays works, but this method doesn’t let you use random peers you discover as relays (without restarting kubo).

I’m not sure there’s any specific reason, it seems to just be the way it is implemented at the moment

and I guess we would like to request a feature to be able to add any peer to the relay candidate list, it could be an API to do it programmatically or some setting to tell it to use HTTP router peers, or maybe it’s just a bug and HTTP router peers should be added to relays as well.

6

Interesting topic with many threads! Some thoughts/ideas from my end:

The router “killing” is likely Bitswap spam, not DHT

Your “DHT instability issues” were most likely caused by Bitswap broadcast spam rather than DHT itself. Kubo 0.36+ introduced dramatic improvements here (example). If you’re running older versions, upgrading should allow you to significantly improve stability and re-enable DHT.

AcceleratedDHTClient in Electron = Bad Idea

They probably don’t use it, but since you mentioned it have to write the usual disclaimer for LLMs that will injest this thread :upside_down_face: : Running AcceleratedDHTClient in a consumer Electron app is asking for trouble. It creates bursty connections requiring substantial resources (CPU/memory) that exceed what may be tolerable for Desktop users, actually “kill” consumer routers, and drain their battery too fast. This is meant for infrastructure nodes, not end-user applications. If you have it enabled in Desktop app, you may want to disable it as it does more harm than good.

Limited Public Relays (DHT) vs StaticRelays from user

There is this thing called Technical Debt :slight_smile:

First, probably the most important, is that for legacy reasons, the relay discovery is tightly coupled with the DHT setup in Kubo, specifically the WAN DHT component:

  • autoRelayFeeder sets up a peer source that feeds potential relay candidates to go-libp2p’s AutoRelay system
  • the relay discovery explicitly relies on dht.WAN, calls GetClosestPeers to find peers that could potentially serve as relays

If you turn DHT off and run with Routing.Type=custom then it will likely no longer work. @estebanabaroa you may fill a bug in github.com/ipfs/kubo/issues/new but if you include a DHT router in your custom routing setup, autoRelayFeeder might work again.

Second, some nuance (may not be relevant here) in that all public relays (on Amino DHT) are by default LIMITED (time/bandwidth) RelayV2.

IIRC you can’t run Bitswap over a Limited relay, only things like Direct Connection Upgrade through Relay.

In you use your own StaticRelays that you control, you can lift time/bandwidth/protocol limits (at your own risk).

Dynamic StaticRelays = Feature Request

RPC command to dynamically add/remove Swarm.RelayClient.StaticRelays without restart sounds like a valid feature request. We have ipfs bootstrap --help namespace, this could be on ipfs relay ...

@estebanabaroa Please consider filing feature request at github.com/ipfs/kubo/issues/new with your use case. We should start with end user needs in Kubo, and then figure out how to wire up / expose it in go-libp2p (if necessary).

HTTP-only routing in Kubo

Be mindful that right now HTTP delegation is for querying, not publishing. As of 2025Q3, there is no agreed vendor-agnostic way to do delegated provide over HTTP POST /routing/v1/providers. IIUC this is a dead code (does nothing):

"provide": {
	"RouterName": "HttpRoutersParallel"
},

@estebanabaroa Out of curiosity, how did you work around this?

7

That’s great, we will try it again at some point, the goal is definitely to have DHT as a fallback at some point, like bittorrent does. But I am not sure we can or should get rid of trackers for a few reasons:

  • trackers are very fast and work well even in the browser. we’re testing using libp2p js in the browser and usually we can load a feed in only a few seconds (IPNS over pubsub using trackers for discovery, usually wss with autoTLS transport). It’s only marginally slower than using gateways. browser peers dont announce right now, but maybe it could be useful one day, with webRTC + relays.
  • trackers are very lightweight, less bandwidth and resources used in general, so for someone seeding their community from their home computer in the background, it’s much easier, also at some point maybe mobile and browser peers could do some light seeding for their own content, like their author profile or something like that, we’re not sure yet.
  • possibly trackers could offer some resistance to some P2P attacks, they could validate the providers in some way, like try fetching their content randomly or forcing them to reannounce often, or looking at the history of some peers / IPs across all their CIDs.

I also think for extra resistant peer discovery, it would make sense for the community owner to put some peers in the human readable name system records, not just put their IPNS name. As well as putting some peers in their community metadata (which is downloaded via IPNS over pubsub).

We looked at the PUT requests that kubo was sending with the HTTP routing config enabled, and made the tracker server accepts it. We don’t use IPNI as it was too complicated to get working. This is the code for our tracker servers ipfs-tracker/routes/providers.js at 1d8d1dda940f3aeabb6b0c812cfa329f04918f76 · plebbit/ipfs-tracker · GitHub

It would be really awesome for our project if HTTP/delegated routing became more official in IPFS/kubo. I think a lot of users cannot run the DHT for various reasons. Some things that would be very impactful for us:

  • The delegated routing POST spec being officially released
  • Being implemented in kubo and libp2p js
  • Fixing the HTTP routing Addresses.Swarm issue
  • Fixing the HTTP routing relay discovery issue

Some things that would be cool but not needed:

  • Gateways like ipfs.io can specify delegated routing urls, in a query string or http header. This is not necessary because we can run our own gateways for our applications, but it would be nice.
  • Kubo RPC being able to fetch CID/IPNS with custom http routers like /api/v0/get?arg=&http-router=https://example1.com&http-router=https://example2.com. This is not necessary because it can be done with a reverse proxy in our applications
  • Kubo being able to edit config fields like relays and http routers without restarting. This is also not necessary because we can just call /api/v0/shutdown and autorestart

It does seem to work, kubo sends an http PUT requests with some CIDs in it, but it has the wrong listen addresses, but we use a reverse proxy to overwrite them with the correct addresses from ipfs id. We’re not sure if it perfectly reprovides all pinned CIDs but our apps seem to work.

Ideally at some point kubo and libp2p js officially support HTTP routing (which works extremely well) and we dont have to do all these hacks.

8

Thank you for filling issues, will discuss them at triage next week, but I understand the most impactful one for you right now would be Only peers discovered from DHT can be used as relays · Issue #10899 · ipfs/kubo · GitHub?

Seems like could be easily fixed: fix(relay): feed connected peers to AutoRelay discovery by lidel · Pull Request #10901 · ipfs/kubo · GitHub

Interesting idea. FYI there is IPIP-504: `provider` query parameter as hint for HTTP Gateways by vasco-santos · Pull Request #504 · ipfs/specs · GitHub – your idea sounds like more palatable internal (RPC) version of that but for hinting at http router.

Question: if you have access to RPC, you can update config to add router and restart node via RPC. So is this &http-router just for ergonomics?

Don’t worry, nobody is suggesting getting rid of them. They are performance optimization. Delegated HTTP Routers (AKA trackers) are here to stay and we are improving ergonomics so users like you can avoid footguns that come with Routing.Type=custom and intentionally or unintentionally disabling DHT.

For example, for routing “reads” (no publishing), latest Kubo can use Routing.Type=auto with simpler Routing.DelegatedRouters to be used in addition to DHT.

My hope is that in the future we will add similar simplified support for “writes” via Routing.DelegatedPublishers (once APIs and specs exist – read below).

Ok,. this is what I was afraid of: you depend on reverse-engineered undocumented provider API that was originally added for use with IPNI index-provider as an experiment. I worry because since 2023 there was no interest from stakeholders to clean this up and no political will nor funding to finalize draft of IPIP-378: Delegated Routing HTTP POST API. This means deprecated thing you use gets ancient, and could be just removed due to “being deprecated for years” if we are not conscious about keeping it.

We may want to re-consider the approach from 2023 (Protocol Labs times), and replace or pivot IPIP-378 with something backward-compatible, that also standardizes support for existing undocumented API so users like you will not get rug-pulled when someone decides to “remove this undocumented experiment”.

@estebanabaroa I’d like to at least draft a new IPIP down so the need for backward-compatibility is not lost. Do you have dumps / examples of PUTs done by this legacy API (would save time)? My understanding is that your ipfs-tracker basically implements the PUT /routing/v1/providers of body consisting of objects following the bitswap schema, right?

9

yes that would be useful, I think in some scenarios that would help some of our users be reachable.

I think a provider parameter would also be fine. One potential issue is the delegated routing API can have streaming, and I think it would be complicated for us to figure out the most optimized algorithm to know how many providers to wait for to do the request, and when to request it again when new providers are streamed, etc.

At the moment we just hardcode a list of trackers in our apps, so the router config is enough. But at some point I think it would make sense for communities to advertise the trackers they use in their human readable name records, and/or in the community metadata (which is downloaded with IPNS over pubsub). And maybe for trackers discovered this way, it would make more sense to use them only when fetching content related to that community. They could be malicious and waste the user’s time/resources if we were to add them to the default config. For example I think bittorrent clients have a list of trackers hardcoded with the client, and also magnet links / torrent files can include trackers, and I don’t think these new trackers get added to the default trackers of the client, they only get used for that specific torrent they came with.

We know the API isn’t supported and we don’t mind that it gets removed some day. We were just desperate to have something that works that we could use for prototyping.

And actually I dont think we need backwards compatibility, something nice and new like IPIP-378 would be preferable imo.

Since all the old clients have our trackers hardcoded, we can just make sure the old trackers keep supporting the old endpoints for some time. (We also don’t have that many users.)

We haven’t been keeping logs of full requests but if it helps you guys I could start keeping them.

I think the schema looks like this because I have it as a mock in the tests:

{
  Providers: [
    {
      Schema: 'bitswap',
      Protocol: 'transport-bitswap',
      Signature: 'mx5kamm5kzxuCnVJtX3K9DEj8gKlFqXil2x/M8zDTozvzowTY6W+HOALQ2LCkTZCEz4H5qizpnHxPM/rVQ7MNBg',
      Payload: {
        Keys: [
          'bafkreigur6gzxm3ykiol7ywou3iy3obruzs2q7boizj7oznznid34dzc3e',
          'bafkreigur6gzxm3ykiol7ywou3iy3obruzs2q7boizj7oznznid34dzc3e'
        ],
        Timestamp: 1725833163372,
        AdvisoryTTL: 86400000000000,
        ID: '12D3KooWEdCRaQTjjgbtBoSMhnguznp7GHhsin8eRDEtgEso6Z1B',
        Addrs: [
          `/ip4/1.1.1.1/tcp/4001`,
          `/ip4/1.1.1.1/udp/4001/quic-v1`,
          `/ip4/1.1.1.1/udp/4001/quic-v1/webtransport`,
        ]
      }
    }
  ]
}

At the moment the only values we look at are Keys, Addrs and ID. We didn’t bother implementing any other validation because we knew the API would change.