Setup advice - use case

1

Greetings. I am a developer hoping to delve into IPFS with a new project. I have read a lot of the docs and think I am heading down the correct rabbit hole, but I could use some affirmation on my thought process. I hope I have posted in the right forum and that these questions are sensible.

I generally would like to set up at least one (possibly three) nodes on Digital Ocean for my use. Based on my reading I think I will likely use GO-IPFS on Ubuntu VMs. I say three because I have read that retrieval will be faster, the closer my file is to the requestor. So my thought is the build a node on a NY server, a Europe server, and an Asia server with Digital Oceans tools. I would then pin any files for my projects onto all three nodes. Or is that totally unnecessary? Perhaps just one would suffice.

This is the general use case: I and a small group of friends make NFTs and would like to pin the content and metadata files without having the pay for a third-party pinning service (which I have been doing for too long because I haven’t yet just built my own - lol!). The files are mostly, JSON, images (PNG, JPG), and some single-page HTML websites with inline CSS and JS. I want the nodes to serve the content that I pin there and make that content available to the public, but I do not want my nodes to be abused with spam or junk. I am just not sure exactly what to build. I obviously want my content available to the public and am all about being part of the decentralization, but don’t want my nodes full of junk either. Is this possible and if so how do I set that up?

It seems like I need to build a pinning service that I can use to pin items (and perhaps one day offer up to a curated group of other people). I would like to be able to pin with JS code in addition to CLI so I have been looking at the API. I am thinking these three nodes (or maybe just one) with this pinning service should be behind a reverse proxy server with some kind of auth, so that I can cull the group that is able to pin. Does that sound right or is all of the overkill?

Another thing I am not really understanding is the gateway. Gateways are to resolve files locations, but I don’t think I need to build that since I typically offer up a url for a public gateway for retrieval of my content. And I understand that nodes have their own local gateway but I don’t want that open to the public. Its a little foggy here for me on whether I even need to bother with a gateway and if not how to I keep it from becoming some other problem for me.

Phew! Thanks for hanging in there to read this. I welcome any comments or feedback on these ideas. I am a competent developer but also do not know everything - especially in the IPFS space - Lol! So I could use any sound advice that anyone who has been down this road can offer. Please go easy on me if an of what I wrote doesn’t make sense. Lol!

Cheers!

1 Like
2

I would start with one.

If you’re mostly fetching over the HTTP gateway, then, you can run an nginx proxy in front and/or run a CDN in front.

But it sounds like you mainly want to keep files online and eg use IPFS.io.

Note that performance is best effort through public gateways. Another area where multiple locations won’t necessarily help.

1 Like
3

Hi Boris, Thanks for your kind response. What you say makes sense. I have actually already started down the road of following this guide, which seems to be in line with what you are describing. The complete guide to hosting a public IPFS gateway ¡ GitHub
I think that performance is not terribly important for retrieval in my case, but I still wonder how much worse it is on the ipfs.io gateway. I typically give the urls for my data as CIDs prepended with “https://ipfs.io/ipfs/”, which seems to load fairly quickly for the most part. I am still quite a noob here though so if anything I am throwing out there seems not-quite-right, I would love to have it pointed out to me. Thanks again. Dave

4

Glad that you’ve gotten started!

You’re asking the public gateway to serve up bandwidth and your files. This is fine (for now, see later link), but any complaints of “it’s slow” means you need to run your own gateway and cover the setup and costs.

The wovin team is working on some self hosting code setup here ucan-host-ipfs - Open Collective

@gotjoshua @tennox are the Wovin team and may have some thoughts on how this applies. “Just” persisting files is a pretty good use case.

1 Like
5

Very helpful. Thank you. These concepts I read all seem within grasp but in reality there are so many details to make all of this work optimally. I appreciate the info.

1 Like
6

Using a mash up of several guides and some document diving (and several trips to the snack cabinet), I was able to get the node deployed to Digital Ocean. Things started going awry when I attempted to run a service daemon to keep things going when I disconnected. I used the code described here: kubo/misc/systemd/ipfs.service at master ¡ ipfs/kubo ¡ GitHub, adjusting for my directories of course.

It seems fine after reboot when I type this:

systemctl status ipfs.service

I see this:

     Loaded: loaded (/etc/systemd/system/ipfs.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-02-09 04:37:21 UTC; 5min ago
       Docs: https://docs.ipfs.tech/
   Main PID: 627 (ipfs)
      Tasks: 8 (limit: 1100)
     Memory: 125.2M (swap max: 0B)
        CPU: 23.208s
     CGroup: /system.slice/ipfs.service
             └─627 /usr/local/bin/ipfs daemon --init --migrate

Feb 09 04:37:21 dgsc-ipfs ipfs[627]: Swarm announcing /ip4/159.203.66.220/tcp/4001
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: Swarm announcing /ip4/159.203.66.220/udp/4001/quic-v1
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: Swarm announcing /ip4/159.203.66.220/udp/4001/quic-v1/webtransport/certhash/uEiDI>
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: Swarm announcing /ip6/::1/tcp/4001
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: Swarm announcing /ip6/::1/udp/4001/quic-v1
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: Swarm announcing /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDIdQg70X3bH0D>
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: RPC API server listening on /ip4/127.0.0.1/tcp/5001
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: WebUI: http://127.0.0.1:5001/webui
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: Gateway server listening on /ip4/127.0.0.1/tcp/8080
Feb 09 04:37:21 dgsc-ipfs ipfs[627]: Daemon is ready

But when I type this:

ipfs swarm peers

I see this:
Error: this action must be run in online mode, try running 'ipfs daemon' first

And then when I type this
ipfs daemon

I get this

Kubo version: 0.26.0
Repo version: 15
System version: amd64/linux
Golang version: go1.21.6
2024/02/09 04:45:28 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
Swarm listening on /ip4/10.108.0.3/tcp/4001
Swarm listening on /ip4/10.17.0.5/tcp/4001
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/159.203.66.220/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/159.203.66.220/tcp/4001
Swarm announcing /ip6/::1/tcp/4001

Error: serveHTTPApi: manet.Listen(/ip4/127.0.0.1/tcp/5001) failed: listen tcp4 127.0.0.1:5001: bind: address already in use

None of this was going before I made the service daemon. And I am not very experienced with linux to make matters worse. Last week I got this going on a RASPI 3b with no problems.

If anyone can shine some light on this I would appreciate it. Alas, I think it is more of a Linux problem than an IPFS problem, but I am not certain. Any advice would be appreciated.

Of course, I still have hurdles once this is done.

Dave

1 Like
7

I was able to get it working again by disabling and enabling the daemon in systemctl. Now to work on making the node on the droplet available outside the droplet.

8

Hi there :slight_smile:
I’ll add my two cents:

If you run kubo nodes (the new name for go-ipfs) that is already a ‘pinning service’, and you can use the Admin RPC API to upload/pin content.

That’s exactly what we’re building with ucan-store-proxy: a way to allow people (via UCAN) to store content on your node! :wink:

but kubo also supports basic/bearer auth - if you trust the other users, as there is only one user so they have full admin api access.

Kubo will not store anything (* except DHT buckets, but that won’t be a problem afaik) you don’t pin or request via the gateway - and the gateway you can either disable or set up to only serve content you have pinned.

ipfs.io or cloudflare-ipfs.com work fine in my experience, but for more serious use-cases you might want to either self-host your gateways (and only serve pinned content) or rent custom gateways (filebase, CF, fleek, …)

That’s just the CLI not finding the API, check that your IPFS_API env is at up correctly or use --api /ipv4/...

If you need explanations or links or support, don’t hesitate to reach out, now I’m just quick-answering from my phone ;p

9

Super helpful! And much appreciated. I will start digging back into these topics soon and I am sure to have more questions. I may be wrong but based on some digging, I believe my problems with the online/offline also have something to do with NAT on my Digital Ocean setup not handling the ip addresses well - I am researching it on the tip too. But I will definitely check it all over. The other insights you provided are also exactly the kind of things I needed to have confirmed, but that are often not overtly stated in docs etc. So thank you, it is all quite helpful.

Cheers,
Dave

1 Like
10

I figured out the service deamon problem and now when I reboot and then use

ipfs id

I see this :

{
	"ID": "12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
	"PublicKey": "CAESIDW0fZRH3JmsGnL2HpHHr9P8EW5101fYQtCbzUyyhRn1",
	"Addresses": [
		"/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
		"/ip4/127.0.0.1/udp/4001/quic-v1/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
		"/ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiDIdQg70X3bH0DXHWXEiRivOvjwfqLZf4cvLCWSDmvaYg/certhash/uEiAlhLATQLcPCwWZ8nqj7uAj1bdQKr1Fjibp0I58EgMUKw/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
		"/ip4/159.203.66.220/tcp/4001/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
		"/ip4/159.203.66.220/udp/4001/quic-v1/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
		"/ip4/159.203.66.220/udp/4001/quic-v1/webtransport/certhash/uEiDIdQg70X3bH0DXHWXEiRivOvjwfqLZf4cvLCWSDmvaYg/certhash/uEiAlhLATQLcPCwWZ8nqj7uAj1bdQKr1Fjibp0I58EgMUKw/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
		"/ip6/::1/tcp/4001/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
		"/ip6/::1/udp/4001/quic-v1/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU",
		"/ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDIdQg70X3bH0DXHWXEiRivOvjwfqLZf4cvLCWSDmvaYg/certhash/uEiAlhLATQLcPCwWZ8nqj7uAj1bdQKr1Fjibp0I58EgMUKw/p2p/12D3KooWDS1QMms2Y1c2vWjm6NaXdK8Ftn7nXLnzJZWxLZuaR1AU"
	],
	"AgentVersion": "kubo/0.26.0/",
	"Protocols": [
		"/ipfs/bitswap",
		"/ipfs/bitswap/1.0.0",
		"/ipfs/bitswap/1.1.0",
		"/ipfs/bitswap/1.2.0",
		"/ipfs/id/1.0.0",
		"/ipfs/id/push/1.0.0",
		"/ipfs/kad/1.0.0",
		"/ipfs/lan/kad/1.0.0",
		"/ipfs/ping/1.0.0",
		"/libp2p/autonat/1.0.0",
		"/libp2p/circuit/relay/0.2.0/hop",
		"/libp2p/circuit/relay/0.2.0/stop",
		"/libp2p/dcutr",
		"/x/"
	]
}

All of that seems great, right?

So, lets pretend that the Digital Ocean droplet that is running my node is located at ip address: 159.203.66.220

And there is text file with this cid pinned to my node:
QmXnVqt4nuU1W1XuBbYVsoj4a16NCdp1Yw6n35JZBiaaic

If everything is right, shouldn’t I be able to see it from a browser when I navigate to:

http://159.203.66.220/ipfs/QmXnVqt4nuU1W1XuBbYVsoj4a16NCdp1Yw6n35JZBiaaic

or

http://159.203.66.220:8080/ipfs/QmXnVqt4nuU1W1XuBbYVsoj4a16NCdp1Yw6n35JZBiaaic

or

https://ipfs.io/ipfs/http://159.203.66.220/ipfs/QmXnVqt4nuU1W1XuBbYVsoj4a16NCdp1Yw6n35JZBiaaic

And shouldn’t I be able to see the webUI when I navigate to

http://159.203.66.220:5001/webUI

or

http://159.203.66.220/webUI

For some reason none of this is possible. I have tried this both with and without a NGINX config on my Digital Ocean Droplet (a config that worked with another webapp just fine). I am at a loss. But I think it is a network problem and not a node problem. Maybe I have staring at this too long.

11

I finally got everything spinning with this guide:

The only thing I did differently than the guide was that I used a newer version of ipfs in step 2.

tar -xvzf kubo_v0.26.0_linux-amd64.tar.gz

Now I just need to get this configured the way want it. Oh yeah, and it needs a certificate.

2 Likes
12

Hi Tennox, I promise I am not ignoring you. I am just now seeing that my “reply” to your very helpful message was actually in the main thread and not a direct reply to you. Apologies. I am clearly not too adept with these forums. Thanks for your help and tips. I seem to have gotten most of this working.

1 Like
13

Haha, no worries!

Glad you figured out your network situation! :partying_face: