How to retrieve content uploaded via Helia using the IPFS gateway?

Tried using Helia today:

import { createHelia } from 'helia'
import { strings } from '@helia/strings'

const helia = await createHelia()
const s = strings(helia)

console.log(await s.add('quick test djkfjdkfjd'));

It returned this hash: bafkreidoqdzmahiqe75jsfuu3cm3wdrmeipvjtz4b7l5xpuaidnlt65f3a

And yet, the IPFS gateway returns an error when looking it up: https://ipfs.io/ipfs/bafkreidoqdzmahiqe75jsfuu3cm3wdrmeipvjtz4b7l5xpuaidnlt65f3a

Why is that? How can this be fixed?

1 Like

Two things:

  1. You need to ensure the Helia node is running so peers can fetch data from it
  2. You need to pin data to prevent it from being GC’ed
1 Like

I have exactly the same Issue.

I created an easy example to test this issue: helia_.html and also created an issue on GitHub

If you want data (a specific CID) to persist in the IPFS network, at least one node needs to:

  1. have a persistent blockstore (via never shutting down, or saving to disk, etc…)
  2. pin that content (so it’s not garbage-collected)
  3. provide that content.

Another way to look at this (wording from @Jorropo)

When the last node that held a copy of your content stops providing the content it won’t be accessible anymore.
Reasons this can happen:

  • The node shutdown.
  • The node crashed.
  • The node has not pinned the data and it was removed by garbage collection.
  • The node has trouble keeping up with announcing in the network.

If you are connected, your node has the content, and it can connect to peers, but you can’t leave that particular node running permanently, here are some quick & easy ways to solve this problem (while we all wait on browser connectivity transports such as webtransport/webrtc to become more reliable and ubiquitous):

  • Use a pinning service (https://web3.storage, estuary, pinata, infura, gw3.io, etc) that will run an IPFS node for you that pins (prevents GC) the content.
  • Use your own backend/persistent IPFS node (Helia/Kubo/etc), and ensure it supports webtransport/webrtc or other, yet-unknown, browser-compatible transports.

In the future, if you have trouble figuring out why you can’t access content on the IPFS network, some great troubleshooting tools can be found at Diagnostic tools | IPFS Docs. With pl-diagnose and ipfs-check being the best, respectively, for troubleshooting networking/content-discovery issues.


Related Helia issues

2 Likes
  • Use your own backend/persistent IPFS node (Helia/Kubo/etc), and ensure it supports webtransport/webrtc or other, yet-unknown, browser-compatible transports.

for me that is the problem, my persistent IPFS node running Helia on the browser is not accesible through any gateway and tools do not show the node ID nor CID (but CID are well formed). So I believe examples run a single local node in browser but do not connect to the main IPFS network. Some other node ID are shown in logs on browser. Not sure if those nodes are local or public. I believe there are some browser issues as already test it with same issues on firefox and brave. I am still working on debugging my setup and code to ensure is not an error from my side.

Maybe I am wrong but the expected behavior is that user open up a webpage, run the Helia node, upload user’s content, pin it and content is available across all IPFS network from any client as long as user has open and running the page in the browser.

@eccker

user open up a webpage, run the Helia node, upload user’s content, pin it and content is available across all IPFS network from any client as long as user has open and running the page in the browser.

That’s a fair expectation if all the items previously mentioned are completed. Have you used pl-diagnose to debug problems? Do you have a code example that’s easy to run and test? (on codesandbox, stackblitz, codepen, etc…?)

I extended our script-tag example from the helia-examples repo that lives on codesandbox to:

  1. Use the latest versions of packages (from npm not script tags)
  2. show multiaddrs which you can attempt to dial

You can find it at https://codesandbox.io/p/sandbox/helia-script-tag-forked-3q8y35

With this node running from codesandbox in the browser, I could fetch new, unique content created by my Helia browser node, from a local Kubo node. Below are the steps I took, you will need 3 separate windows – one browser window and two terminals(shell prompts) – to complete this exercise.


An exercise in retrieving content from Helia

In your browser

  1. Open up https://codesandbox.io/p/sandbox/helia-script-tag-forked-3q8y35 and ensure the helia node is running and generates some Multiaddr(s) to connect to
  2. After you have some multiaddrs created, Copy the multiaddrs listed on the page, and paste to a text file somewhere. For reference in these instructions, we’ll call them commands.txt
    1. Prefix EACH line with npx go-ipfs swarm connect , they should look something like npx go-ipfs swarm connect /ip4/###.###.###.###/udp/...
  3. run the following script in your browser console:
 const textEncoder = new TextEncoder()
 const cid = await heliaFs.addBytes(textEncoder.encode('test-123-YYYY-MM-DD-Other-unique-text-stuff-to-ensure-unique-cid')) 
// If you don't create a unique CID, you are not guaranteed to retrieve the data from your Helia node.
 console.log('cid', cid.toString()) 
 // note this CID because you will need it.
 for await (const event of helia.libp2p.services.dht.provide(cid)) {
   console.log(event)
}
  1. Ensure you copy the CID logged to your console

In a terminal on your machine

  1. run the following: npx go-ipfs daemon

In a separate terminal

  1. paste the commands from commands.txt and run them.
    1. You should see a message output like connect $YOUR_HELIA_PEER_ID success. If you do not, repeat step 2 from the browser section above to ensure you have the latest multiaddrs, and try again.
  2. run npx go-ipfs cat $CID_YOU_COPIED_FROM_THE_BROWSER_CONSOLE
  3. You should see the unique text string you created in your browser.

At this point, you should also be able to get the same unique text string by visiting any IPFS gateway:

Note: if you’re not using a chromium based browser with webtransport support, you probably won’t ever get successful connections, discover more peers, and will likely never be able to generate multiaddrs. e.g. in Safari, the browser Helia node does not connect to anything and because of that, cannot discover new peers. I only get the following errors in the console: WebSocket connection to 'wss://ny5.bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa' failed: The operation couldn’t be completed. Protocol error

2 Likes

I feel I should re-iterate that we still need to wait on browser connectivity transports such as webtransport/webrtc to become more reliable and ubiquitous for your particular expectation to be the norm that “just works” instead of something that requires the stars to align.

Until then, creating the IPFS content in the browser will usually require pinning your content using a backend (provider service or your own node).

Thanks, with your described process I was able to retrieve the pinned content from by browser. But is it possible to do this without having to explicitly connecting the swarm via a command on the interested nodes? I would expect Helia to let other Peers and the DHT know which content it has pinned right? Does this need some specialized libp2p config apart from using the normal createHelia() command? My connected number of Peers is also Pretty low. On average maybe 15 Peers. Could that be the issue, as it is taking to long for the content to be published around the network?

But is it possible to do this without having to explicitly connecting the swarm via a command on the interested nodes?

Yep, it the viable browser-supported transports just haven’t fully propagated throughout the IPFS network yet, but we’re close. One particular issue you may want to keep an eye on is Enable WebRTC Transport · Issue #9724 · ipfs/kubo · GitHub

I would expect Helia to let other Peers and the DHT know which content it has pinned right?

To check if Helia is letting the DHT know that it has content, you can verify with the “Is my content on the DHT?” portion of PL Diagnose

Does this need some specialized libp2p config apart from using the normal createHelia() command?

It shouldn’t.

The default libp2p config for a Helia node is https://github.com/ipfs/helia/blob/9139f30e857f4e247202e0d113027190a04892ba/packages/helia/src/utils/libp2p-defaults.browser.ts. It looks like it’s set to client mode by default, and you will need to change it to ‘server’ mode to do what you want… but acting as a DHT server is pretty expensive and may not be reliable in the browser. Please let us know if you run into any issues. This is incorrect, see below comment by Adin. Thanks Adin

My connected number of Peers is also Pretty low. On average maybe 15 Peers. Could that be the issue, as it is taking to long for the content to be published around the network?

This could certainly be an issue. In past testing, I needed to discover somewhere around 600 peers before I could connect reliably to a decent set of peers, and that took anywhere from 30min to 2 hours.


The main thing to take away here is that until more browser transports are enabled throughout the network (webrtc + webtransports + wss), browser nodes are best for fetching. If you’re running a service that depends on getting content from one browser node to another, you should either be using a pinning service provider, or running your own backend service.

2 Likes

The discussion here is great and I completely agree the usecase needs to get satisfied. To ensure we have a pointer in Github for tracking this work across Helia, libp2p, Kubo, etc. I created [🏆 Golden path scenario] Browser-authored contend retrievable by another machine through the ipfs.io gateway directly · Issue #182 · ipfs/helia · GitHub . Let’s still use this discussion post as the main discussion zone.

2 Likes

Being a DHT server → contributing to the DHT network health and storing of records
Being a DHT client → using the DHT to do things like advertise and discover content (sometimes called Put/Get of Provider Records)

So there’s no reason a browser should benefit from being a DHT server, and ultimately given the low uptime of browser nodes (e.g. a tab closing killing the node) it’s likely unhelpful to the network as well and so shouldn’t be enabled.

2 Likes

If there was interest in browser-DHT-servers, what would a reasonable uptime minimum be for network health?

Given background service worker scopes, and persistence options like the File System Access API (incl origin-limited OPFS), I think most desktop browsers could provide at least 22 hours of uptime as to hit the ipfs defualt provider record republish interval…

as for connectivity – thanks to all the work from folks committing to services like: holepunch ( direct connection upgrade through relay), /webtransport, /websocket, /webrtc, /relay; it seems like we could actually get to a future where services need not require supernodes (in the future supernode will probably mean anything not a browser :stuck_out_tongue: ) to thrive

cc @helia

1 Like

@SgtPooki, thank you for providing this example. However, I was unable to successfully retrieve data from Helia using the example. Could you verify if the example is working with the current date?

@reshetovdenis Hey there,

I can’t seem to get codesandbox example to work well anymore. I think that the connectivity demo has gotten a little too big for its britches. I don’t recommend running the codesandbox examples much because it’s hard to edit (being based off of a browser-script-tag example, and having all code in one file) and doesn’t seem to work consistently inside codesandbox.

I have moved most of my testing experiments and env improvements to GitHub - SgtPooki/helia-playground: just testing stuff with helia, which now has a “fetch content” button that opens a slide-out util that I would recommend trying if you’re trying to confirm that Helia can fetch content.

The UI on helia-playground some further fleshing out, but the code is much cleaner and more manageable from a maintainer’s perspective. NOTE: It won’t render content like a gateway, but you can look for a blocks:get:blockstore:put message in the util log for confirmation that the content for your CID was retrieved. This will be improved, as I have time, to work more similarly to GitHub - SgtPooki/helia-service-worker-gateway

If you’re having trouble fetching content from Helia, but can fetch it with Kubo or other IPFS implementation, I would try out https://pl-diagnose.on.fleek.co/ to debug what’s happening.

With that said, can you share the CID you’re using and the exact steps, or code, you’re following so I can try to help out? Reproducible examples make it much quicker and easier to help :slight_smile:

1 Like

Hi @SgtPooki !
Thank you for providing a Github version. I tried it and failed in getting a data from Helia with local ipfs daemon running on the same machina as Helia. Could you help me to get a working solution?

Thanks for the document @reshetovdenis! That helps me understand a bit more about what you’re doing and what could be the problem.

First,

authoring content in the browser and retrieving from the IPFS network is currently blocked by two main epics: [🏆 Golden path scenario] Browser-authored content retrievable by another machine through the ipfs.io gateway directly · Issue #182 · ipfs/helia · GitHub & [🏆 Golden path scenario] Browser-authored content retrievable through the ipfs.io gateway via a pinning service · Issue #256 · ipfs/helia · GitHub. Until we find a path forward for those via methods discussed in those issues, or get the POST/PUT IPFS spec agreed upon and merged, this will continue to be a problem for Helia.

For now, Helia in the browser should be used only for retrieval of IPFS content.

With regard to the retrieval of content from other IPFS nodes with Helia in the browser, we recently pushed some changes to the @next tag of Helia that you can test out today. This work was part of 255 and tracked in Update Helia to fallback to trustless HTTP gateways · Issue #274 · ipfs/helia · GitHub. You can see a rough design doc regarding the changes that were implemented at Create design doc for "helia with trustless gateway HTTP support" · Issue #272 · ipfs/helia · GitHub

Second,

The reason browser-authored content doesn’t work well right now has been documented in many discussions around this forum, in [🏆 Golden path scenario] Browser-authored content retrievable by another machine through the ipfs.io gateway directly · Issue #182 · ipfs/helia · GitHub, and elsewhere, but the summary is:

Browser Helia nodes cannot be dialed confidently by backend IPFS nodes.

The primary causes of this are a variation of the following:

  1. Browsers don’t have TCP and can’t listen on any transport except webrtc
  2. Most backend IPFS nodes don’t have webrtc support
  3. WebTransport isn’t supported well enough on the network: browser-nodes that are known (thanks to circuit-relay nodes) cannot be dialed because the backend nodes “don’t speak WebTransport”, and DCUtR doesn’t kick off.

[🏆 Golden path scenario] Browser-authored content retrievable through the ipfs.io gateway via a pinning service · Issue #256 · ipfs/helia · GitHub covers the old solution for this, but the lack of updated pinning service providers makes this path troublesome… more information in feat: create helia-remote-pinning example by SgtPooki · Pull Request #147 · ipfs-examples/helia-examples · GitHub


So, what should you do, today to unblock yourself?

  1. You could get yourself unblocked today by using kubo-rpc-client to PUT to a backend node you have Kubo RPC API access to, that can host that content on the IPFS network
  2. You could implement a w3.storage API client (or other IPFS pinning service client) for directly uploading content to their servers, so it’s available on the IPFS network

We understand that Helia users are in a rough spot right now for publishing content from the browser, and we’re working on it, but we are prioritizing content-retrieval first.

Hi all, I am trying to build a demo with Waku and IPFS and was hoping to use Helia to provide content and download it on another browser instance. After few days of trying I finished reading this thread realizing it is not possible:D

Will definitely closely watch the mentioned GH Issues as this would be super cool use case:)