Cannot preload data from ipfs-js

[pjt]

Hello,

I try to preload some data on IPFS through preload nodes and retrieve them from an IPFS gateway.
This used to work (with some minor exceptions of one or another preload node being periodically down) but it is no longer the case whatever preload node is used.
Are public preload (node[0-3].preload.ipfs.io) still ready to accept preload requests?

My code looks like this:

const { randomBytes } = require("crypto");
const Debug = require("debug");
const fetch = require("node-fetch");

Debug.enable("ipfs:preload*");

const IPFS_GATEWAY_URL = "https://ipfs.io";

const preloadRandomBytes = async () => {
  const ipfs = await import("ipfs").then(({ create }) => create());
  const buffer = Buffer.from(randomBytes(8));
  const result = await ipfs.add(buffer, { preload: true });
  const { cid } = result;
  console.log("cid:", cid.toString());
  console.log(
    `fetching ${cid.toString()} from IPFS gateway ${IPFS_GATEWAY_URL}`
  );
  await fetch(`${IPFS_GATEWAY_URL}/ipfs/${cid.toString()}`).then((res) => {
    if (!res.ok) throw Error(`IPFS gateway bad response: ${res.status}`);
    console.log(`found ${cid.toString()} on IPFS gateway`);
  });
  await ipfs.stop();
};

preloadRandomBytes();

[danieln]

While the preload nodes are still running and should be working, we are planning on shutting them down

[Tracking Issue] Shutdown PL-hosted delegated and preload nodes

Can you please share the error log that you’re getting or a link to a running reproduction?

[pjt]

Hello thank you for the quick answer and the head up on pomegrenate project.

To give a little more context, I use ipfs-js in an isomorphic library (node & browser) to provide a simple way for users to upload some data that require global availability for a short while.
As I understand big changes are coming in the next few months and we should prepare to migrate from ipfs-js.
Meanwhile, I still need to provide the “upload to IPFS” feature but the default preload mechanism does not work anymore for me.
Running my own preload to have more control over it could be an option but I’m not sure it helps.

Here are the steps to reproduce the observed issue with node 16 & npm@8

mkdir ipfs-debug
cd ipfs-debug/
npm i ipfs node-fetch@2
touch index.js

paste this code in index.js

const { randomBytes } = require("crypto");
const Debug = require("debug");
const fetch = require("node-fetch");

Debug.enable("ipfs:preload*");

const IPFS_GATEWAY_URL = "https://ipfs.io";

const preloadRandomBytes = async () => {
  const ipfs = await import("ipfs").then(({ create }) => create());
  const buffer = Buffer.from(randomBytes(8));
  const result = await ipfs.add(buffer, { preload: true });
  const { cid } = result;
  console.log("cid:", cid.toString());
  console.log(
    `fetching ${cid.toString()} from IPFS gateway ${IPFS_GATEWAY_URL}`
  );
  await fetch(`${IPFS_GATEWAY_URL}/ipfs/${cid.toString()}`).then((res) => {
    if (!res.ok) throw Error(`IPFS gateway bad response: ${res.status}`);
    console.log(`found ${cid.toString()} on IPFS gateway`);
  });
  await ipfs.stop();
};

preloadRandomBytes();

and run it

node index.js

here are my logs:

Swarm listening on /ip4/127.0.0.1/tcp/4002/p2p/QmQUNt8uu1J7ohvZMYhY7gm5omAaXvSnxQyqzmjFxyPLrA
Swarm listening on /ip4/192.168.1.103/tcp/4002/p2p/QmQUNt8uu1J7ohvZMYhY7gm5omAaXvSnxQyqzmjFxyPLrA
Swarm listening on /ip4/172.240.51.1/tcp/4002/p2p/QmQUNt8uu1J7ohvZMYhY7gm5omAaXvSnxQyqzmjFxyPLrA
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/p2p/QmQUNt8uu1J7ohvZMYhY7gm5omAaXvSnxQyqzmjFxyPLrA
  ipfs:preload https://node3.preload.ipfs.io/api/v0/refs?r=true&arg=QmRL41zVdzdMAUDHQL1pQdkLtLv7HtVaaSoTnj2eF1Any4 +0ms
cid: QmRL41zVdzdMAUDHQL1pQdkLtLv7HtVaaSoTnj2eF1Any4
fetching QmRL41zVdzdMAUDHQL1pQdkLtLv7HtVaaSoTnj2eF1Any4 from IPFS gateway https://ipfs.io
/home/pierre/ipfs-debug/index.js:19
    if (!res.ok) throw Error(`IPFS gateway bad response: ${res.status}`);
                       ^

Error: IPFS gateway bad response: 504
    at /home/pierre/ipfs-debug/index.js:19:24
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async preloadRandomBytes (/home/pierre/ipfs-debug/index.js:18:3)

we can see:

• the ipfs.add(buffer, { preload: true }) promise resolves
• fetching the data from ipfs.io gateway ends with an HTTP 504 while the local node is still running

NB:

• I know propagation in the network may be slow, so I tried retries on the fetch part with no luck
• I also tried different IPFS gateways

[danieln]

@pjt

Meanwhile, I still need to provide the “upload to IPFS” feature but the default preload mechanism does not work anymore for me.

One path that might be worth exploring is running a Kubo node and exposing the add endpoint of the Kubo RPC API (note that to expose only that endpoint you will likely need a reverse proxy like nginx or a middleware HTTP server that only passes through requests to that endpoint).

On the client side, you could use the GitHub - ipfs/js-kubo-rpc-client: A client library for the Kubo RPC API to add files explicitly and then you’ll have better control over the whole uploading to IPFS.

In fact, this approach is similar to what running a preload node would be. See this reply on what a preload is:

Differences between bootstraps, delegates and preload nodes?

This is a great question and a common source of confusion. The first thing to note is that you can use a single Kubo node with a TLS certificate for Bootstrap, Delegates, and preload. So here’s what they are for: Bootstrap is list is a list of peers with which the IPFS daemon learns about other peers on the network. IPFS comes with a default list of trusted peers, but you are free to modify the list. Delegates are nodes that expose the /api/v0/dht/* endpoints of the Kubo RPC API for delega…

[pjt]

Thank you for your answer,

One of my requirements is to keep things as much decentralized (or a least distributed) as possible. That’s why I would prefer running a preload node (exposing the refs api) and adding it to the preload nodes list of ipfs-js to provide a fallback rather than directly uploading content on a node via /api/v0/add, I may run and trust this node but devs using my lib may not.

In the linked post you said

Preload are nodes that expose the /api/v1/refs endpoint […]

I thought the refs endpoint was under the v0 api (/api/v0/refs) is it a typo?

Edit: kubo + js-kubo-rpc-client works fine for direct upload

[danieln]

@pjt

That’s why I would prefer running a preload node (exposing the refs api) and adding it to the preload nodes list of ipfs-js to provide a fallback rather than directly uploading content on a node via /api/v0/add, I may run and trust this node but devs using my lib may not.

That’s understandable, though beware that preload nodes will go down. The preload pattern is not sustainable for two reasons:

• Uploads are implicit — when you can the refs endpoint, you rely on the preload node pulling from a browser tab. Because browsers impose resource and network restrictions, pulling can fail even if you just change to another tab.
• There’s no support for an authentication mechanism, so they can often be abused.

@pjt

I thought the refs endpoint was under the v0 api (/api/v0/refs) is it a typo?

You’re absolutely right! That was a typo.

[abhisheksharma]

It sounds like you may be experiencing an issue with loading data from IPFS using the ipfs-js library. There are a few potential causes for this issue:

1. The IPFS node you are trying to connect to may be down or unavailable.
2. The data you are trying to load may not exist on the IPFS network.
3. There may be an issue with your code that is preventing the data from being loaded properly.

To troubleshoot the issue, you can try the following:

1. Verify that the IPFS node you are connecting to is up and running.
2. Check that the data you are trying to load is available on the IPFS network by using a tool like the IPFS explorer.
3. Verify that your code is using the correct syntax and parameters when trying to load the data from IPFS.
4. Ensure that your application has the correct permissions to access the IPFS network
5. Try to reproduce the problem with other libraries or tools

It would be helpful if you can provide more information about your code, the environment, and the error message you receive, so I can give you a more accurate solution.

[pjt]

@danieln

Uploads are implicit — when you can the refs endpoint, you rely on the preload node pulling from a browser tab. Because browsers impose resource and network restrictions, pulling can fail even if you just change to another tab.

Yes, that’s a good point, I used to fetch the preloaded document in the browser tab from another IPFS gateway to ensure the upload was successful.

@danieln

There’s no support for an authentication mechanism, so they can often be abused.

Isn’t a publicly exposed /api/v0/add endpoint of a kubo node subject to similar abuse?

I’m currently running for tests propose a kubo:v0.18.0-rc.2 with wss enabled and full API exposed

• uploading data to that node works with both direct add and preload
• I can fetch the uploaded content from the gateway of another node my company hosts (that node is a go-ipfs:v0.9)
• I can fetch the content from the pinata gateway
• but I can’t fetch the uploaded content from the public https://ipfs.io gateway, should I do something (other than opening 4001) to make my node discoverable by the node exposing the public gateway?

Erratum:

• With prelaod method, I sometimes get the uploaded document from the ipfs.io public gateway. I observed a call to a delegate node, when this call fails the public gateway also fails to serve the uploaded document. So the problem might not come from preload nodes?

[danieln]

@pjt

Isn’t a publicly exposed /api/v0/add endpoint of a kubo node subject to similar abuse?

Yeah it is which is why we warn strongly against exposing the full RPC API — unlike refs, the add endpoint will also pin the file.

@pjt

• but I can’t fetch the uploaded content from the public https://ipfs.io gateway, should I do something (other than opening 4001) to make my node discoverable by the node exposing the public gateway?

That’s strange. I would try a number of gateways to find out whether it’s gateway specific.