IPFS Gateway that fetches from specifc peers but serve to all peers in the network

,
1

Is there a way to create a network of IPFS Nodes that only fetch files from each other i.e., some explicitly mentioned peers, but serve the files to all the peers on the network ?

All the existing ways like a private network with swarm key, removing bootstrap nodes list and adding peers to peerlist, turning noFetch on will stop the peers from fetching the files from other peers and only serving files present locally/internal network but it would also restrict the nodes from serving files to other peers which we need in our use case

2

If you explicitly wanted an HTTP gateway, you could use an IPFS private network of nodes with the data you want to have, and have them serve over HTTP (Kubo comes with a built-in gateway you can make public with a reverse proxy). That way only the CIDs you put into your network are served.

Would that work?

3

If I understand you correctly, add your nodes to the bootstrap section in the config file. I have two nodes that connect via a bootstrap entry. They are always connected. I can access what I upload from either node without waiting for normal propagation. Yet the nodes still work as normal nodes.

Node 1 should have Node 2 in the bootstrap section and vice versa.

1 Like
4

This doesn’t work, since if you only put your peers in the bootstraped list, your CID would be inaccessible to other nodes and gateway. For instance ipfs.io, dlink.web

5

I still want the nodes to connect to IPFS.io and cloudflare nodes for instance so any public gateway can serve those files.

6

Yes, it does work. Nowhere did I say to remove the other bootstrap nodes. I only said to add your nodes.

I have been running with this setup for over six months with no issues.

If you want a private IPFS network, then yes, remove the other bootstrap nodes.

– Jamey

7

If I add other bootstrap nodes, my gateway would serve random CIDs as well, and if I enable NoFetch that would make the CIDs(the ones i actually. wanna serve) on other nodes (private network) inaccessible.

8

Kubo (IPFS node implementation which ships with gateway, and also provides data to other peers over bitswap), has Gateway.NoFetch configuration option which disables data fetch being triggered by gateway requests, making gateway only provide data that is present in local datastore. This gets you 80% there.

@Man-Jain If you want to solve problem today, you could solve remaining 20% by either:

  • (A: userland) Writing some HTTP middleware and put it in front of IPFS gateway that runs with Gateway.NoFetch set to true.

    • Intercept requested gateway URL, try fetching response from local gateway, if it fails, retry and manually fetch/preload requested data from other nodes you own.
    • If any of your nodes had the data, your middleware imported it to local node and can now retry request to local gateway once again – it will now succeed for this and future requests, because a copy of data was replicated and cached in local datastore.

  • (B: updatream) Implementing new Kubo feature named Gateway.OnlyPeeredFetch which works similar to existing Gateway.NoFetch but will fetch missing data if one of Peering.Peers has it.

    • This is not a priority for Kubo team, but sounds like a sensible feature, so if someone invests time and submits a PR with tests, review can be prioritized.
2 Likes