Gateway: how to serve only pinned CIDs

Help
1

Hi all,

I’m running an online archive which pins all assets from NFTs minted on a particular marketplace.

To do this I run a server-side app that identifies which CIDs to pin, and for each one uses the /api/v0/pin/add RPC call on a local instance of Kubo, which fetches and pins the CID in a single operation.

This Kubo instance also acts as public gateway, however right now it’s serving all CIDs, even those not pinned. I’d like to only serve pinned content.

It seems like a solution is to use the "NoFetch": true setting for the Gateway, however when I do this, calls to /api/v0/pin/add no longer work, so I am not able to pin new CIDs.

Is there a way to achieve both?

2

What good is a public ipfs gateway if you only allow access to a specific set of data? You’re not having any IPFS benefit at all anymore so why even bother using it at all? Rather, just download all these CIDs to a folder and host that folder using plain simple static nginx.

You can still use IPFS behind the scene to do your new updates or whatever.

Problem solved, no?

3

Running with Gateway.NoFetch mode in Kubo will support trustless, verifiable response types, which would be missing when hosting static folder with nginx.

Exposing CIDs over HTTP in a way that includes verifiable response types is a positive thing, allows clients to do retrieval over HTTP instead (or in addition to Bitswap) and enables everyone to benefit from HTTP caching and CDNs.

While not as mature as providing data over libp2p and Bitswap, the IPFS & HTTP: The Blueprint for Radical Interoperability Talk from recent IPFS Camp elaborates bit more on current work towards “direct retrieval over HTTP”, and connecting clients with such “HTTP providers” (probably better term than calling them “limited gateways”).

@ktorn Which version are you running? The functionality you described is supported, works fine at least in the latest Kubo I had at hand (0.30.0-rc1).

These are not related settings / features, IPFS Gateway API and Kubo-specific, internal RPC API run on different ports:

You can run Gateway with NoFetch:true, limiting what can be retrieved from your Gateway (:8080) only to “blessed” CIDs that are already in your repository, and at the same time use internal ipfs pin add CLI or :5001/api/v0/pin/add HTTP RPC API calls to preload only the data you care.

$ ipfs config --json Gateway.NoFetch true # + restart your ipfs daemon to apply
$ ipfs pin add CID # pin CID that should be provided by your gateway port
1 Like
4

I’m running the ipfs/kubo docker image

/ # ipfs --version
ipfs version 0.29.0

with NoFetch = false

/ # curl -X POST "http://ipfs:5001/api/v0/pin/add?arg=Qmd8w32fPqa1niDQpQqooxznhB5qjsFdmw713pcDcaV6VW"
{"Pins":["Qmd8w32fPqa1niDQpQqooxznhB5qjsFdmw713pcDcaV6VW"]}

with NoFetch = true the exact same curl command with same CID just hangs.

5

I was not able to reproduce with v0.29.0 either, the CID you used above pins fine with NoFetch.

Perhaps the problem is related to something specific to your setup?

  • Try upgrading to 0.30.0-rc1 to eliminate already fixed libp2p and bitswap bugs. If you run both gateway and storage instances, make sure to upgrade both.
  • Check if your node’s connectivity is correctly set up, namely expose ports defined in Addresses.Swarm, and allow incoming connections.
  • TIP: when you are trying to debug things, and want to ask for help, share your full ipfs config show + CLI / env that you used for running Docker. Without it it is hard to reason about your setup.
1 Like
7

Port 4001 was not exposed on the firewall, good call.

But oddly, even after exposing it, a pin HTTP request was hanging, even with NoFetch = false.

I rebooted the docker container and, without making any other changes, the pin request worked.

Then I changed the config to NoFetch = true, restarted the container, and it also worked.

Very odd, but at at least I’m glad I can pin content with NoFetch = true enabled.

Thanks!