All-in-one Docker image with IPFS node best practices

Ecosystem
,
21

The custom domains is probably the least self-hosty, you’re right.

A user would have at least one domain and create mygatewayaddress.com. Turning on subdomain support means that lots of sites could look like bafy1234.mygatewayaddress.com.

I didn’t think the _DNSLink stuff was hard but I was also thinking that Cloudron (for example) has let’s encrypt cert generation and would be done outside this app.

I think host gets passed to Kubo and Kubo looks it up, and nginx doesn’t have to do anything?

@walkah can you describe how this works?

1 Like
22

I think fly.io’s pricing has changed since. Based on my experience, Kubo needed a minimum of 512MB of RAM So it was around $10/month.(Fly.io Resource Pricing · Fly Docs)

Screenshot 2024-01-08 at 10.55.37 am
Screenshot 2024-01-08 at 10.55.37 am1538×994 66.5 KB

1 Like
23

Have a look at this issue and feel free to share any thoughts:

24

When I compared some of these services a while back to host IPFS stuff, they didn’t support hosting UDP services which is needed for QUIC, WebTransport, and WebRTC.
This may have changed, but it’s worth considering depending on the use-case.

1 Like
25

This is a great question and a topic with a lot of nuance.
@tennox To keep this on topic, would you mind opening a new topic for this?

2 Likes
26

=> Moved to here

Repo

I set up a basic git repo (RFC). (if anyone wants write access, tell me :wink: )

Nginx caching

Set up a basic test on a branch, but it seems to me there are no performance gains (hurray, kubo gateway - you are as efficient as an nginx cache). Further testing / test cases could change that :face_with_monocle: (or I misconfigured something :thinking: )

1 Like
27

yeah, afaik there’s nothing required at the nginx/proxy layer - kubo itself will resolve DNSLinks (I always assumed based on Host: header but I guess I’ve never actually looked at that code).

1 Like
28

Yep, correct. Somehow I only thought about static hosting & nginx, and didn’t think of kubo :sweat_smile: Nevermind then :ok_hand:

Ucan Store Proxy

Started a PoC for a service that checks if a UCAN if it authorizes to store/pin, and forwards that request to the Kubo RPC API (and later maybe other pinning services :thinking:).

Does that sound like a decent strategy?

2 Likes
29

First PoC success :tada:

‘Happy path’ PoC-complete:

  1. Deployed to Akash (my cloud of choice for this test & costs 3$/month - let’s see :yum:), adding my note3 did in TRUSTED_DIDS env (which not checked yet)
  2. Created a UCAN & used this code in our browser app to PUT a CAR file via the proxy to the Kubo node
  3. retrieved it via the Kubo Gateway (demo link)

Open questions

I’ve added some feature ideas & questions to the repo - what I deem relevant to this thread:

  1. Is hosting a public gateway a good idea?
    • Can/will it be abused? (probably)
    • is it a problem? (e.g. fetching data and no GC - full disk, hosting, but also responsibility for content)
    • Option 1: Gateway only serves pinned/stored data (NoFetch)
    • Option 2: Use Gateway through ucan proxy (bit finnicky, but might make some use-cases possible)
    • Option 3: Disable Gateway, and fetch via own Kubo or public GW only… kinda defeats the purpose :sweat_smile:
  2. Implement pinning API on the proxy? (or even as an extra storage?)
    • Instead of “just” API for uploading (e.g. via CAR-file) - I thought we might as well add pinning possibility. (PoC already implemented) This allows to pin someone else’s data on “your” server.
    • but how much of the pinning API is relevant/useful

Would be happy about feedback :slight_smile:
(to the general strategy of this concept, but also specific parts)

3 Likes
30

Woo hoo!

Y’all should create a Project in the Wovin OpenCollective so we can support this.

And supporter votes on features should be taken into consideration :wink:

2 Likes
32

:heart_eyes:

:partying_face:

33

rather here? or on the issues in gitlab?

34

At your discretion :sweat_smile:
I guess for the general concept & topic here, and for specific features in the issues.
Also not sure if part of this discussion would better belong in a separate thread. :thinking: (anyone feeling spammed? :yum: )

New repos & Opencollective

Consolidated repos to new org:

And we have set up our opencollective sub-project
in case anyone want’s to help make it happen on a financial level :wink:

Regarding “Is hosting a public gateway a good idea?”

Option 2: Use gateway through ucan proxy (bit finnicky, but might make some use-cases possible)

I thought: as we probably need a reverse proxy for Letsencrypt SSL anyways - we could add a forward_auth middleware in front of kubo.

But I’m not sure how different self-hosting situations would look and if that would make sense… @boris any idea on that?

1 Like
35

I’d love to support this financially. There are a couple of different targets.

Maybe @danieln or someone else could take a look at getting this running on FlyIO and see what it needs — those systems typically run their own nginx networking layer and then reverse proxy to the app that’s running.

Cloudron is the platform is one of the self hosting platforms I’d love to see this running on. It’s got some specific requirements:

3 Likes
36

I just came across this guide: Enhancing IPFS Performance in Kubernetes Environments :: Terminal Thoughts which I thought might be relevant for this thread

2 Likes
37

Some containers for various ipfs peers can be found here - GitHub - boyroywax/ipfs-liftoff: Tools & Examples for IPFS, Libp2p, OrbitDB, Bacalhua, and more.

1 Like
38

Thanks for sharing! Had a quick look and these are mostly packaged as Docker Compose. Do you have any thoughts on other deployment config best practices?

39

I have deployed on rpi k3s, k8s, digitalocean, azure, aws, and gke. apologies for the late reply. let me know what u help with …

40

Less about the tools — more about config and setup of Kubo and other components. Nginx, caching settings, how badbits is configured, how it’s setup to use / auth APIs etc